I had one living under my backyard neighbor's shed for many years. I watched it get really fat on the surrounding neighbors' vegetation, but it never bothered my stuff. It would cross my yard within 15 feet of me and we seemed to have a mutual respect going.
Then, last summer, it had a litter of pups and as they grew, they had the whole neighborhood mad. I lost several things and I knew they would continue multiplying so out came my pellet rifle. I took out 3 of them and a neighbor managed to club one with a baseball bat. He smoked their hole in his yard with some kind of chemical he got at a farmer's supply. That seemed to take care of the rest.
Now, we still have one living under that same shed. Just like the last one, it eats all of the neighbors' stuff, but it leaves mine alone. Thus, I leave it alone. I think I've figured out why. I have kids and they are always playing with their friends in our yard. I am constantly out there gardening and stuff, so there is alot of human activity. On top of that, I have a cat. The cat and groundhog never bother each other, but they watch each other very carefully. The cat often stays out at night.
I have no fencing around my garden, but I do grow alot of stinky stuff like sage, oregano, onions, garlic, basil, dill, marigolds, etc. There are alot of rabbits and squirrels around along with the occasional coon. No problems, except my seedlings sometimes get robbed in the spring. That might be birds.
Maybe it's the cat.