There's a pretty obvious and rather common imbalance going on: Charlotte is responsible for making sure Emma behaves and grows up at least something decently (since there are no parents, apparently). So, she naturally needs to be strict and make Emma do things that are necessary but that Emma wouldn't want to do, due to being just a little child. Aoyagi, however, only sees Emma occasionally and can spoil her as much as he wants, seeing how he bears no responsibility. This will make Emma view Aoyagi like some kind of goodwill vending machine who will never make her do anything she doesn't want to do and even takes her side against Charlotte.
What comes to Aoyagi himself, he must be feeling lonely every now and then, living alone, and he also keeps his distance from most of the classmates for his chuuni reasons. I have no doubt he simply enjoys spending time with someone.