In this story, the reason for their prejudice is religious, as alluded to in the prince's dialogue. Kind of like the mark of Ham being used to justify chattel slavery or any religion with women - doesn't have to be fear, it just needs to make life more convenient for people with more raw power...