OK, here goes.
So I will work the mats and thin them out . Should I do that now or in the spring?
First off, no mat maintenance until after the spring thaws. You want more mass over the winter to keep the corms from freezing too badly (although it sounds like you only get stem damage, which is terrific.) Mulch 'em in well and you'll be fine.
How much should I thin ie how far apart should the plants be? Should I remove the smaller, younger plants or take out the large old ones to make room for the newer ones?
Generally, leave the mother and two daughters, ie the largest that hasn't fruited yet, an intermediately-sized one, and one smaller one, with preference for "sword" type pups - those have very thin leaves to begin with but are taller. "Water" pups, which have broader leaves to begin with, are poor bearers and generally less healthy plants. Take anything you've removed and plant it elsewhere, and put the water pups in the compost or give them away.
Should they be fertilized now? What NPK ratio do they like? Should I spread compost after cleaning them up?
No feeding if you're heading into frost. Generally, bananas can be treated like very big grass - balanced NPK is a good idea, but when you start to see flag leaves (the small leaf that comes just before the flower head) up the K dramatically. So if you're using something like 11-11-11 (which is generally recommended) when you see the flag, switch to 5-10-20 or something similar. That will aid fruit development and ensure that the highest possible number of female flowers set fruit.
Absolutely top-dress with compost, though. That's never a bad idea.
What about the brown foliage? I have been leaving it on the plant until spring thinking the brown leaves would help protect the plant from cold.
That's a good idea. If you want to keep the pseudostems more insulated, you can wrap burlap and frost cloth around them as well. This might even be enough to keep them from mooshing out on you.