Saffron Plums?

[JessicaMSH]

I have 2 "saffron plum" trees that I think may be infected or diseased. There is a scaley texture all over the trunks and branches. The fruit grows in a large quantity, then falls off the branch before it ripens or gets big enough to pick (usually golf ball sized). Any suggestions on pruning or treating this tree?

Thanks,
Jess
P.S. When I first bought the property, the tree had plums the size and color of peaches. We haven't seen that since. We are also coming out of a drought as well. Just a little more info!