Do a close inspection. See if there are small brown with striped caterpillars in the head. They can take over a head of broccoli, -ick factor. If you see a few, can cut off the head, cut into sections, soak in cold salted water. If too far gone, may end up in disposal or compost!
It looks like aphids. You can blast them off with a strong jet of plain water. Check the leaves, top and bottom, as well as nearby plants. Ladybugs and lacewings are extremely effective controls, but blasting them off will keep them at bay. Keep tabs on them every couple of days at least.
In my experience this kind of damage is typical of slugs/or whatever initial damage, it attracts slugs. Once this kind of partial spoilage occurs, it attracts all kinds of pests, damaged area will keep spreading and the entire head will be ruined. I've had better success salvaging the head by harvesting immediately and cutting out any part that I wouldn't want to eat.
In any case, to me, the flower buds look like they are getting ready to open. This can happen, too, due to the damaged cells triggering the "We're dying! Hurry up and mature!" plant hormones.
For me broccoli, and for that matter, all things in the cabbage family are very hard to grow because the bugs so like it. Seems every kind of insect critter known loves to eat it. Not to mention snails and slugs. To be successful you need to do some serious, diligent insect protection. Good luck!
I usually have the same issue with cabbage family plants......this year is an exception, in the row next to my pole beans, wedged between a row of dill and onions, is where I have my kale planted.....my kale usually gets decimated by little green caterpillars.....this year, absolutely no damage, I have huge bushes of kale.....wondering if it's due to the proximity to the dill and onions????
Have you had a lot of rain lately and wet weather? The combination can cause soft rot of the broccoli head. While I would not discount insect feeding, slugs and caterpillars would also be making holes in the leaves. The leaves in your picture don't look so "holey". What do the leaves look like?