Unfortunately, there is a tendency to place the veg garden at the edge of a small property. It would help if your property sits higher above the grade/slope than the neighbor's. If at all possible, it's probably better to maintain even a small buffer space of ornamentals or have a solid fence separating the properties.
Believe me, I would design some of my own garden spaces differently if I could, though I'm making small changes year to year.
One good way is to change your perspective and consider edible landscaping -- I.e. Don't limit yourself to a dedicated vegetable garden.