I think you are ok. Go ahead and plant. Have a great garden.
Potting soil? Have you ever looked at the ingredients in that bagged stuff? It varies, but mostly it contains peat, vermiculite, perlite, sawdust. None of which are very high in nutrients, but do serve as a place to anchor roots.
As far as soils go, anchoring the roots is the first requirement. Water retention and providing nutrients are important soil functions too.
Manure and compost are good sources of nitrogen and other plant nutrition.
Natural soils are based on clay, silt and sand. They also contain water, air, organic matter, chemicals, and a host of living organisms.