how does the other person feel about all of this? do they feel as excited and drawn to keep charlie as you do? talk to them and see if they would let you simply have him!
how great would that be insteed of moving him around every two weeks mayb set up a day for all 3 of you to have a little play day and mayb send pics offten or even videos that way they can still feel connected too! try to talk to them and see what they want as well
good luck!!