Not entirely clear to me that manure is mineral deficient. In fact, I've seen studies with poultry and steer manure that show that it is rich source of Cu, Zn, Bo, K, and Ca, as well as lots of N (though perhaps not in immediately useful forms). A lot of trace minerals are considered necessary for agricultural animal supplements, and these largely get passed with the rest. Also, organic material (compost, composted manure, etc.) holds moisture wonderfully and is very permeable, such that runoff is less of an issue.
It is possible that soil structure that includes sand, pebbles, and other mineral particulates is better for mechanically supporting root systems, in that it is heavier and the constituents can lock together better. That is, in a medium that is too "soft" and lightweight, a large plant could basically just tip over.
One thing to watch out for in a heavily manured plot is salt. In many parts of the country, bagged manure is pretty salty. The composting process tends to dry out the manure, and that just concentrates whatever salt was in it. But salt leaches out pretty readily in the course of a season. That's one advantage of using ground soil. Any leachable bad stuff has already been leached out of it. Another bad thing is ammonia, though well composted manure should have most of that removed.
So in my view, gardening in pure composted manure probably isn't a great idea, though you might be able to get away with it.