Thanks again for all the replies, everyone!
, I tried what you said. When I flicked the rosemary above a white sheet of paper, a lot of tiny, tiny flecks landed on the paper, but they didn't move at all. They did leave a brownish smear when rubbed. A couple whitish fly-type bugs (some dead, some alive) also fell after a few vigorous flicks. Lastly, I did see some webbing, but it could've been cat hair.
Not sure what to make of all this. Would it hurt to assume spider mites and treat the plants accordingly?
Regarding your question about lighting, the plants are on my apartment balcony, but the balcony gets A LOT of sun, so I've moved all the plants under the balcony railing. The rosemary was yellowing and browning on the other side of the balcony, so I thought that was probably a bad sign. All the plants still get a lot of sun, but they are no longer directly in it for most of the day.