Yes, you are. Those might be something like thrips. Do you see tiny gnats flying around? If you do, then the bugs in the soil might be the larvae of fungus gnats. Both of these pests require very moist soil to live.
IMO, African violets should be allowed to dry a bit between waterings. Not so much that they wilt, but enough so the top half-inch or so of soil is dry.