Couldn't you learn to like it?
Some landscape designers deliberately plant creeping thyme or other ground covery things like that into the cracks in pavers. The contrast between the brick color and the green is very nice. I liked the look of your picture. Just go over it with a weed whacker every once in awhile so that it doesn't get tall... Saves a lot of work!