Hello and welcome to the forum. Moving the tree back and forth, especially inside to out, is not a good idea. Since you are able and seem open to an outdoor tree this is always my first choice. A spot that receives morning sun and afternoon shade is good place to start although a Chinese Elm can take considerably more sun once healthy.
Lots of leaves started to grow again but this time the leaves develop a white/grey powder on their surface, (top and bottom). They then curl up and fall off. The leaves don't dry up or change colour.
This sounds like Powdery Mildew. If you leave it inside eliminate the humidity tray and see if this helps. You may not need it at all during the summer, but do save it for use during the winter if you intend to keep it inside then.
If you move it outside again don't bother with the tray, I don't think they do much good outside anyway. Either way remove all fallen leaves and trash them.
Since this is a small plant you can probably wipe most of the Mildew off with damp paper towel, then make sure to get the tree in a situation where it gets some air movement to dry the leaves. BTW, I never mist Chinese Elms (except perhaps as cuttings) as they are susceptible to another fungal disease known as Black Spot. Keeping the foliage dry is important.
It might also be time to look into some sort of anti-fungal spray. One that has been mentioned on this forum is a milk spray. There are also traditional Copper based sprays. I have never used either so be careful, especially with the Copper based spray as some plants are sensitive to certain treatments.
The leaves on the lower branches however don't seem to be doing this, but they are hugely bigger than the leaves when i first got the tree.
The aforementioned Black Spot also seems to effect new foliage. Older leaves can naturally get larger than newer ones. Another possibility is that the plant is not getting enough light and the leaves are growing larger in an attempt to compensate.
Just go slow and steady with the tree don't try to overcompensate with excessive watering or fertilization, just good cultural practices should bring it around. Read this for tips on watering bonsai.