When I wacky rig I have been using a 1/0 - 3/0 Eagle Claw 249WA with good hookup ratio and few lost worms. These are fairly light wire hooks so hauling a big fish out of the weeds could bend the hook. (Sadly, I have not encountered this "problem" yet this season.)
I tried the Gamakatsu Finesse Weedless Wide Gap hooks but I kept hooking the fish deep and I lost a whole bunch of worms because the O rings kept sliding off the hook.
This weekend I ran across some Mustad W37754R wide gap weedless hooks that have intrigued me. I purchased 2 packs in 1/0 and will be testing them next time I go out.
I think you answered your own question, hook size! You are not breaking off so it isn't a line issue. Bump up the hook size and use a good quality hook and you should be ok!