@dimox already answered that for you, in the manga we see literally the "elite", women that especifically are trained to not get crazy arround guys, and even with that they all falling to Sou's charisma because he's that nice.
The few normal women we saw wanted to never let him outside home to...