As this is my first attempt ever at any kind of gardening, I hope my question doesn't sound too ignorant!

So I'm trying my hand at container gardening and started all my seeds in little peat pots. The seeds sprouted and began growing wonderfully! The time came for me to harden off the plants and move them to their more permanent homes (in various containers) on my deck. So I did what the peat pot manufacturer said to do - I planted everything - peat pot and all! Again, all seemed to be going well... until my plants all (except oregano) seemed to stop growing.

I didn't pay much mind, but did keep a close eye on them. Then, maybe two months later, I decided to move some plants to a larger container and put some in container to give away. When I dug them out of their current home I noticed that all the peat pots still seemed to be in tact - they didn't appear to have degraded at all! Not only that, but there seemed to be a yellowish-greenish fuzz growing on the outside of the peat pots! So I'm freaking out a little thinking that it's mold... Also, there were very few roots poking through the little pots, so I'm thinking that the pots are stunting the plant's growth - could this be correct?
Anyhoo, this seems to be the case for my jalapenos, chives, cherry tomatos, and scallions. Can anyone help me understand what's happening here?
Sorry for such a long post! And thank you for ANY info you can give!
Karmachelle