Bumblebees usually give you plenty of warning -- and the head-butting that you've mentioned before -- before attacking I believe. I've never been stung by a bumblebee though I have
been head butted on occasion. I once accidentally dropped a banner out of the upstairs window and startled a bumblebee, which zoomed at and headbutted my DD, who was about 4 yrs old at the time, watching me from below. It didn't sting her, though she was plenty scared.
Maybe you could put up a fence around the bird house to mark off the safe distance and keep people away (and protected). At the end of the season, when it gets cold enough that they'll hopefully stay quiet in the chill early mornings, you could either sacrifice the birdhouse and move it elsewhere, or try to pry the nest out and put it elsewhere.
I was too cheap to let them have the box and the nest was a small one so that the Winter contingent was only the Queen and 4 or 5 followers, but I scraped out a paper wasp nest during the winter and the poor things did nothing to stop me. I felt bad because I doubt they survived the move, but I couldn't leave them in there.
ETA -- Ted I can't imagine being stung in the head. I was stung TWICE in the calf by yellow jackets and that hurt and swelled up badly enough. I'm allergic to mosquito bites and get large welts so I might be more susceptible to bee/wasp stings as well, however. Hope that the stings heal up soon.
ETA2 -- Gary, some people are allergic to preservatives or even containers used for pharmaceuticals and the simplest treatment can potentially make them considerably worse rather than better.