An instance where the fall/winter/spring freeze here helps -- I leave the birhouses up year around because sometimes they can be depth of winter shelter for the birds. But if no"bird" is using them, I can also go around checking for wasps nest, and knock them right out along with the cold-immobile queen and her drones.
They are usually small nests -- 1/2 shell walnut to no bigger than my small fist size at most (only IF the wasps had nested in it since the summer and didn't get evicted)
I welcome or tolerate wasps in most other not inconvenient locations, so I feel no compunction in evicting them when they choose to nest where they are not wanted -- utility boxes, hose storage, doorways, and birhouses are some of the places they should not be nesting. Most of the wasps here are not aggressive and we get along fine in the garden, but Yellow Jackets are mean and get another degree of intolerance though.
Generally, by the time I see the weathered wasp's nest again laying in the ground, something -- mice, etc. have been enjoying the unexpected find and the nest is chewed up and empty. I consider that just part of the natural cycle.
Sometime in early spring, I check them final time before the birds would start nesting, make sure there are no wasps nest and remove spiders for ones with openings large enough for not just house wrens (house wrens' favorite food is spiders and they use spider egg sacks for nest lining, so I feel it's OK to leave them in there) and dust the floor and side cracks with DE to make sure ants don't get any uppity ideas about high rise living during the spring thaw. This is also when any needed repair work is taken care of.
Or hang gourds after drilling a few holes which only lasts one season but have worked before. 
