i.m.o. i think it could be a couple things. like already mentioned, smell... i always use hand sanitizer before and after. they know i'm not a mouse that way 
second, i'm not sure about yours, but i have a pair of w.s. brooksi that are cage aggresive. if they see something moving in the cage they WILL nail it. once i have them outside and am holding them... they are tame as all get out. they have never once struck outside the cage. i used to feed in seperate containers, but once i got into the multiple rack thing that got to be a major pain.
as far as feeding, everyone has their own opinion. mine is, i have to agree that your baby could probably benefit from some more mice. I FEED smaller mice, 2-3 at a time, just large enough to leave a small lump. i feed twice a week. by the next feeding day they have digested the previous meal and it has become a pile of poo, which is the goal for me. SMALLER meals more often are easier to digest. feed more often and a greater #. by doing this they take in more nutrients and grow faster. doing this they don't have as long of a lethargic period where they just lay around and digest. that is My prefered method. to each their own....
i would encourage you to stiil handle it on a regular basis. enough to know you but not so much as to stress it.
jonathan