I don't go soaking in bleach water, etc. but I do inspect them when they are empty of nesting birds during the cold months for wasp nests and ant occupation because either will prevent the birds from accepting the nest box. To do that, you do have to take out the nesting materials to some extent. If there are ants, I evict them then "bomb" the interior with horticultural grade diatomaceous earth.
One year, house sparrows took over one of the nest boxes and filled it with trash -- I mean candy wrappers, bits of tattered plastic grocery bags, orange hay baling string, etc. -- I don't even know where they found them all
A house wren won back the nest box and immediately emptied it of all the garbage. I was astounded by the amount of trash I found under the tree -- in fact, at first I thought the neighbor's kids for inexplicable reason decided to play a prank and trashed my yard.
Sometimes, in cleaning out the previous nest, you will find things that seem unsuited/dangerous for the baby birds. I tsk, tsk, and remember when my kids were babies and I had to "childproof" the house... And wonder what those parent birds were thinking!