I buy the ones with adjustable waists it works for us!

Sorry no answers here but maybe you could modify the larger pants. Maybe just run a small stitch left and right of the waist of the larger size 3 and 4 pants so they fit the waist; or sew a small ‘margin’ from waist line to leg, on the left and right, so that they are made narrower and might fit better (at least until he grows and you can undo it).

Haha, i have this same problem ever since he was a babe, especially the bottoms. I usually have to add elastic into the waist of all his pants. Or find those pants with the adjustable waist. Also, mine loves belts (hard for him to undo to do to the toilet though).