My biggest concern is that you don't know what's
under there

(ghosts and gnomes?)
-sorry a bit of gardening humor there

-
At the very least, pull up or open up several spots along the perimeter and in the middle and dig up some samples and dig down and see what kind of soil goes how deep.
If it's all lovely stuff, sure, plant away. But more likely, it would be best to turn and amend the pretty much dead lifeless fill soil before attempting to grow anything or you'll be disappointed.
I have to say I'm also concerned that in an attempt to keep this area pristine, the previous owner may have heavily applied herbicides and pesticides. A test grow with bean seeds might yield some useful information as beans are said to be sensitive to certain herbicides and chemicals.
Ultimately, you may want to consider having the soil tested first....