The Holocaust disagrees with you. The systematic genocide of other people happens for religious reasons too - look at the Middle East today. India's caste system existed for at least 3,000 years. And you cannot control the oppressed when you are not despotic and wanton to a degree because a paper tiger will eventually be beset upon.And I find this depiction of irrational and illogical violence to be unrealistically over the top because even slaves are valuable. On the other hand even evil is rational to a degree. You cannot control the oppressed when you are this despotic and wanton because there's zero gain (aka naked survival) for them to fall in line under these circumstance. Hence why there wouldn't be any black haired left if this is their "normalcy". It's bad writing.
The fact that mana counts for actual power in this world helps explain why the black haired cannot easily fix their own situation. Unless they show up with a skill like Rick's, they will never be strong enough.