I don't know what to tell you about the critters. They're hungry, and you're growing yummy fruit.
The California Rare Fruit Growers organization offers this information, and does address the problem of fruit drop, which I gather is not uncommon.
"Isn't it enough to see that a garden is beautiful without having to believe that there are fairies at the bottom of it too?" - Douglas Adams