You don't say where you live so I don't know what the environmental infulences are. Here's some ideas from this Texas site.
Why There Are So Few Peaches
I earlier said that I expected to have less than a 5% crop, when all varieties are averaged together--I now think the outlook is much more dismal, possibly less than 1%. Much of what little fruit we counted as still being on the tree is simply not growing as it should be by now.
Following freezing temperatures on the mornings of March 24th & 25th (when we used a helicopter to try to moderate the temperature), I was still optimistic about our chances for a fairly good peach crop, in spite of considerable losses in some varieties. It slowly became evident that this would not be the case. The cause is not so much the freeze as it is the drought last fall and winter, and, most importantly, the lack of adequate chilling over the winter months. Many of our varieties survived the freeze because they were not yet in full bloom, but their bloom was delayed because they had not received sufficient chilling. Without sufficient chilling, they were slow to leaf out--some of them still have poor foliage! Without good leaf growth, the small peaches that looked like they would make a crop are not developing.
You haven't mentioned if the leaves are stunted or darker then usual. If so, take a look here.