In a way Ceres was more of a father to the children than even their actual fathers. He was there for them, he took care of them, and he did it all without a complaint because that’s what their mothers wanted. He did it for them, both their moms and the kids. He was friends with them sure but he’s mentally far older than them so he probably only ever saw them as kids.
He was always attracted to their mothers though, both their bodies and their personalities. So it only makes sense that they would be the first people he would return to, given he was doing all of this for them after all. You better bet your ass he never expected this turn of events, no doubt he’s the happiest he’s ever been because of this.
It’s important to stress that he didn’t marry, and then proceed to sleep with all of their mothers out of spite, he didn’t do it to seek revenge. He did it because he genuinely loves them and wants them to be happy, if anything it’s their own mothers who want to get some payback in place of Ceres.
He was never angry over the whole situation, not because it wasn’t unfair, but because he never cared for it to begin with. He was doing it as a way to protect them, and if they felt that they no longer needed his protection, then he gladly stepped back and parted ways. He was protecting them on behalf of their mothers, so in an odd twist of irony, it was that very same desire to keep their kids safe and happy, did they in turn become safe and happy themselves.
Were it not for that request, Ceres likely would have simply stayed in the village. He most certainly never would have allowed Shizuko to be sold into slavery, he probably would have kicked the ass of her abusive husband. Honestly there is a chance that regardless of what happened, whether he left the village or not, Ceres would have wound up in a similar situation to the one he’s in simply by virtue of the events that transpired regardless.