People forget because Sou’s family, and class, are relatively tame. They don’t want to be hated by him. But his mother and Sumire said they were prepared to be hated for his protection. And as far as we know, that’s how the Native Sou was before the current one reincarnated over.
Reality is outside those strict places like the Guard School and the hospital, kidnapping and reverse rape is common. And the average family prioritizes the physical wellbeing of the son over the emotional one. Sou’s family only caved in because the Guard School had a 0% Incident record.