As Granny used to say, "To each his own, said Charles as he kissed the cow."
Personally, I think used organic-based cat litter is no more gross than chicken, cow or horse poop, which are lauded far and wide for their benefits to compost pile and garden.
I make a separate pile, though, for kitty, and it's not used in the vegetable garden, just to play it safe.