It is a really bad time to be growing broccoli, but yours is being eaten alive by slugs. You will never see them in the daytime, but come out at night with a flashlight and I bet you will find a bunch.
Type slugs or slug control into the Search the Forum Keyword box and you will find a lot written here about them. What has worked best for me is diatomaceous earth and putting broken eggshells around the base of the plant.
But given that there is hardly anything left of those and it is the wrong season anyway, I'd just pull them and plant something else there -- and put the DE down (slugs love broccoli, but they aren't too picky about what they eat).
As Kisal noted, you can replant later. For a fall crop, you could put broccoli seed down in Aug. By the time the broccoli would be getting ready to make heads, the weather will be cooler. Last year I planted broccoli seed in mid-October. It sprouted but didn't get very big by the time frosts came. But given that we had a mild winter, it over-wintered (I did not even cover it) and started growing again in late winter and gave me a great early spring harvest.