Those sticks may be shrubs that die back every winter and then are wonderful again in spring and summer. They don't look like hydrangeas, but could be something along those lines. Can you rock one of those sticks back and forth, and does it then come up very easily with not very much root attached? Then they are probably dead and not dormant. They could also be the old stems of annuals that get a little woody, like sunflowers, also dead. Maybe hollyhocks which are biennial. I'm just guessing here, but if you haven't seen the yard in summer you don't really know what they are. Possibly the neighbors would be able to tell you.
As far as the neighbors plants spilling over into your yard, the ordinances vary by municipality. You probably have the right to trim back anything that is over your fence line or lot line, with the likely exception of pruning something so severely that you would kill it. So if a spruce tree grew on the lot line, you couldn't shave off a third of the branches down through the trunk, thereby killing the tree just because it was over the property line. You'd have to consult with the neighbor, possibly get a variance. But viney, weedy growth, or wayward branches are fair game for you to trim.
Getting a big tree out with so little property almost certainly means you will need a permit, and a licensed tree service will be your best bet. If you are absolutely sure you have the tools, the strength, the manpower, the disposal, and the means to take out a tree, go ahead. They just have a way of doing the unexpected, like wiping out a fence or coming down on power lines.
Congratulations on your new home.