Slaves are slaves because they deserve their position? Because they have some innate quality that fated them to become slaves?
First, that's a BS excuse to prevent guilt and feel superior as they don't want to admit they too might become slaves under the wrong circumstances.
Second, if slaves are fated and deserving of slavery... by their own logic, does becoming some ogre's food also count as the fate they deserve?