If you want to use garden "dirt", you'll need to mix it with something so that the soil is not too dense and allows good drainage. I like to use sand -- about 1 part sand to 3 parts soil, but people often use Perlite. I generally push aside a leaf pile and use the soil underneath (similar to a0c8c's "under the bushes") Under a woodpile is also good. If you have compost, mix some of that in and you have an excellent soil mix. Other good additions are bits of "rotting" logs.
You might want to test your soil mix first. Fill a planting pot with it, tamp it down, then water. If the water doesn't seep right in but puddles at the top, the soil mix is too dense.
Make sure to moisten the soil mix before using to uppot your plants. It should feel like a wrung out sponge. If you squeeze some in your hand, it should hold it's shape then crumble.
Don't be surprised if you have tiny weed seedlings start to grow. Just pluck them out. You'll be surprised how LONG their roots get.