Tell us more about the conditions it is growing in- full sun? have you been fertilizing it? watering? etc.
I can't grow pomegranates where I am, so I don't really know, but I looked up this for you:
Severe fruit drop during the plant's juvenile period (3-5 years) is not uncommon. Mature trees seem to set and hold fruit better than younger trees. Fruit drop is aggravated by practices favoring leafy growth such as over-fertilization and excess watering.
Incidentally that is from an Arizona U. publication and it talks about pomegranates doing well in their hot dry climate. You have hot there in Alabama, but I think not so much dry, so you would really need to be careful about over- watering.