Why don't you post a picture of your plant? It is possible it was mis-identified somewhere along the line, because it doesn't sound like lavender at all. If it is lavender, if you just lightly crush some leaves, it will smell very strongly of lavender. Lavender leaves are pretty distinctive too, almost like needles, looks a lot like rosemary leaves, but the scent is different. "Huge" leaves is NOT lavender. Lavender "leaves" are small, slender, needle like.
So no wonder it was growing so much faster than any lavender I'm familiar with... it isn't lavender.
If you put your lavender seed directly in the ground, maybe these are just weeds that came up in the spot you planted in. Lavender is not easy to start from seed - slow germinating, and needs just the right conditions. I start it from seed but only indoors, on heat mats, under lights, where I can baby it. And still it is slow getting started.
Here's a website with descriptions of all the different lavenders. The underlined name of each variety is a link. If you click on it, it will take you to a picture of that variety.
https://www.mountainvalleygrowers.com/la ... ndtips.htm