There seem to be at least a couple causes of that, that I found (probably there are more)...
A small proportion of our hazelnuts are subject to the attention of a weevil. The nut weevil, like all weevils, is a beetle with a long snout or proboscis which has positively elephantine proportions in the female. She uses it to drill a hole into the young nut as soon as it is set and the shell is still soft. A single egg is laid in the developing kernel, and the hole heals over so there is no sign of infestation. The egg hatches into a grub which feasts on the kernel until the nut falls. The now empty nut has a hole where the young weevil emerged.
This is when a fruit is formed wthout fertilisation by pollen and has no seed. This is fine for grapes but a hazelnut with no nut is just annoying. . . . . The fertilisation process must be done before the leaves unfurl as hazel, in common with most other forest species, is wind-pollinated. Milder winters, and consequent earlier leaf-burst, will reduce the amount of nuts produced the following summer.http://swepstonecoppice.co.uk/Hazel.aspx