Last year I found out that, for the first half of the twentieth century, the site of my and my neighbors' homes was a huge gravel pit. When the homes were built on it around 1950, the pile of rocks, which are the size of those river stones you can buy bagged at Home Depot, was removed and the remaining rocks spread around at a depth of about 6" to 8".
By the time I moved here, in 2006, the rocks were compacted down and covered with a 1" layer of topsoil. To begin a new bed, I have to pound out the rocks with a rock hammer.
The first time I cleared and spaded an area, all I found were two incredibly skinny, pale earthworms. They were tied in knots. I was devastated. Only two earthworms, and they were clearly mutants. A gardening friend told me that earthworms do that just before they die of dehydration. Our summers are dry, and there was no need to water rocks, so my earthworms died or left for greener soil (pun intended).
I use a lot of compost. The next year, I counted perhaps a dozen earthworms when I prepared that bed. The following year, I'll bet I had 50 or 60. Now the bed is teeming with them.
I read somewhere, probably in the Shepard book, that if you can count two dozen earthworms in a spadeful of dirt, you've created an earthworm friendly environment. Although it varies from bed to bed, I can generally exceed that.
You just need to be patient