Well part of the head size depends on the variety of broccoli you are planting. Brassicas have a tendency to bolt in the heat, but you can grow a good crop in the spring if you get them in early and use a fast growing variety.
I myself prefer to plant mine in the fall, I get a better crop and have to deal with fewer insect problems. I normally plant De Cirrco, because it's a fast grower and throws wonderful side shoots once the main head is harvested.
You might want to have your soil tested, it could be a lack of some nutrient that's not allowing them to grow larger.
I'd rather be gardening!