Serissa are finicky about any number of issues, but relative humidity seems to be the big one. Higher humidity in the air rather than the soil seems to be the big issue, but drafts of cool, big changes in lighting and other factors can be problematic.
A few yellow leaves dropping is to be expected this time of year; for the most part your tree appears healthy. A humidifier, spritzing foliage during the day (when it can dry off before night) and keeping it clear of drying sources (away from heat registers, off the top of the TV, etc) will help a lot.
Frankly I think there is a repotting in your spring regimen; that pot seems far too small to me. I like a more nutritive soil for tropicals like Serissa, but that is a matter of style and practices. A wider pot will allow for more moisture retention without sitting soggy and I think will inprove the aesthetic considerably. I'd say half the depth and the width of the canopy, mounded in the middle to accomodate the root mass which I might try to split and shallow some as well.
Just a thought...