Short of bringing them back inside, you can just start fertilizing them. You said the seed soil wasn't feeding them, because it has no nutrients, which is true, but then you said you added compost. But with plants in 3" pots, the compost is not very nutrient dense. It is probably somewhere around 1-1-1 or 2-2-2, compared to synthetic fertilizers which have numbers ranging from 10 to 20. In the garden that doesn't matter, because you can use the compost in shovels full. In the 3" pot it makes a difference. Look around for some good organic fertilizer. If the nutrients are the problem, then bringing them back indoors doesn't make sense.
If you are just using compost, the alternative is to up-pot them again, into like 6" pots, with fresh soil and compost. Or plant them in the garden! I can't believe there aren't things you could be putting in the ground by now.
If you do bring them back indoors, then yes you will have to harden them off all over again.