I don't foresee problems resulting from the earthworm, but I seriously doubt that a cactus, or any other succulent, will be happy for long in typical garden soil. The exception would be if you have extremely sandy soil in your garden. Unless the soil is very coarse/sandy/gravelly/etc., it probably won't drain rapidly enough for cacti and other succulents growing in containers. Your plants could possibly die of root rot, unless you're extremely cautious about overwatering.
The trick will be to give the plants enough water to moisten the entire root system adequately, and then to allow it to dry enough so that rot doesn't set in. The difficulty will be balancing the moisture retained in bottom half of the soil in the container with the dryness of that in the top half.
Heavier soils, which includes most garden soils, remain too wet in the bottom half, while the top half becomes overly dry. That tends to drown the lower part of the root system, causing it to rot, while the top portion dies from lack of moisture. (I know you're not a total novice with plants, so you may very well be up to the challenge.
For new gardeners, or those who lack experience with cacti and succulents, I recommend using a high quality packaged soil designed specifically for plants in this category. For those who have a large number of cacti and other succulents, and have sufficient storage space available, it may be more economical to buy the various components and mix your own growing medium. There are many recipes online, but I like to research the types of soil present in the areas where my plants grow as natives, then try to match my growing medium to that. I no longer have so many succulents to reap the monetary benefits of that, so I just purchase a high quality pre-mixed cactus/succulent soil.
"Isn't it enough to see that a garden is beautiful without having to believe that there are fairies at the bottom of it too?" - Douglas Adams