First off all you should be doing, after the babies are two weeks old, is touching one for one to two seconds. Not holding it for any length of time. While mom might seem ok with it right now they often get upset later on. That behaviour is normal, they aren't very active at that age.
You should have been adding a small amount of kitten food to moms food as soon as the babies were born.