Knowing his wife the fact she witnessed what happened with Ayame Machiko was hired as a minor and the same thing happened she was essentially groomed. And the situation did bug Haru to some level. Haru was even shock Machiko didnt look for other men not following Haru's mothers lifestyle
If you marry someone to protect them you clearly care for them. If he hadn't left I don't think Machiko would have been pawned off to his father by his mother or however just my thoughts