There are several reasons why the females didn't have babies. One, they are too old to breed, or too young. Two, separating them from the male in 3 weeks is stressful and not necessary. It's best to keep one male in with one or more females and leave him in there. Males rarely if ever cause harm to babies, especially when they are the father. Female mice can eat their own young but I never had any problem with this myself but then again I bought my females young and socialized them alot to get used to me. I would hold them and play with them at least till i wound up buying more to breed for my snake.
My mice I kept in large enclosures with an exercise wheel, water and high protein/fat food and often I would watch them play. Sometimes I would hold them but never when they were close to having babies. I never handled the babies or bothered the mothers when less than a week old and try to work cleaning around times the mothers recently gave birth. I have never had mothers that ate their babies except for one time when a mother decided to have her babies minutes before I started taking everybody else out of the cage to clean it. I put her and what babies she had aside to hopefuly not stress her out but she started eating her babies so I put her elsewehre and the babies I just mixed up with other pinkies only a few days older. This worked as once everybody was back together in the now clean cage, the new pinks survived.
Rats are not as prone to eating their own young but take a bit longer for a new colony to get settled enough to breed.I bought a pair of rats to breed and it was nearly two months before the first batch of babies came. I kept two females from that batch and will let them add to the colony.
One of the females I kept is a nice tan hooded white with black eyes. Very pretty.