Sigh.... why can't shitty parents get called out in manga?
Blame...IDK, wider societal expectations? Internalized trauma? Conformity? For that.
They don't see a parent reacting shittily to "their son not being mainly enough" and think it as a negative, they think of it as "teaching them a harsh but necessary lesson". Even if that means kicking them out of the house to live on their own... though I don't remember, does his father actually give him anything to live alone, or was Imaizumi just having to pay everything, besides probably school tuition, from his own pocket before the gals decided to just live with him?