It should be fine! We've had Ronnie for about 2 months now and just went away for 4 days. We left him wth our friends who have a 12 year old son. I was really worried because I felt like we were really just starting to bond with him after a couple of months and I didn't want to cause any setbacks by leaving him for just those few days. Everything was fine though and when we brought him back, I swear he knew he was home! He wandered around sniffing like crazy, I think recognizing the familiar smells. And he was cuddlier than he's ever been that night. I expected just the opposite, but I really think after a few days with an excited little boy, he was happy to get back to his comfort zone
Because hedgehogs are relatively low-maintenance in terms of their daily needs, I wouldn't worry too much about leaving them with a sitter as long as you're really clear with them about his routine and important things like temp, etc.
Having said all that, I agree with hedgielover that it's not a good idea to leave a pet immediately after bringing them home. It's really important to let them settle in, and you want to really get to know them and all their quirks before you let someone else take care of them. Maybe you could take him with you? They're actually pretty easy to travel with, as long as you have a safe warm place where they can stay?