It definitely sounds like a pollination issue to me. I had to hand pollinate for about a week straight after I started getting blooms. The bees were nowhere to be found whereas they were plentiful in the past. I eventually came to the conclusion that the 13 year variety of cicada that hatched this spring scared them off. Those buggers are LOUD. The backyard was covered with them and everytime I would start an outdoor power tool(lawnmower, weedeater etc) they would swarm me. The wife said they were trying to mate with me.
Fortunately their life span is only a few weeks and once they died off en masse the bee's returned. I was quite happy as hand pollination 20+ plants every morning beofre heading off to work was not a ton of fun.
Try Johnny123's tips for hand polliantion. It is sound advice.