Do you think he would retire to stay with his family? Or rather, he would tell the truth and they stay together regardless of his other job?
Because there's nothing that ties them to their current location, and having a reliable group would be beneficial to him.
In other words: AUTHOR-SAN, PLEASE LET THEM BE HAPPY.