I don't think it makes any difference to the health of the plants, really. What you do
want to remove is any flowers. Once they flower, the leaves will die back. At least, that's what happened to mine.
"Isn't it enough to see that a garden is beautiful without having to believe that there are fairies at the bottom of it too?" - Douglas Adams