I think there's some nuance here, but not sure whether or not it's intended by the author:
He didn't abandon them, as in he wanted to cut ties with them for his own selfish motives, but instead chose to run away in order to distance himself from his family in order to protect them.
Imagine a father who's an alcoholic, who knows he's a bad drunk, hopeless, and has a bad reputation: he willingly leaves the family in order to protect them because he sees that there's nothing he can do to help other than to evict himself.
Notice how he doesn't mind Mitsuki visiting him (and appears glad she visited), because she's going out of her way to find him.