I thought so too, but I don't think it's true, because since he was young he also spoke to Uichi, for example, when he told him that he thought Yotsuba was pretty. It would also be something that could be seen from far away.Also, it seems that the sister talks to him as a male when she calls him.