Kinda glad this dude's an actual asshole. He's not doing the overplayed "I was being mean to test you" and he's actually discriminating against slaves.
I always find these characters so stupid. You get rid of the potential threat by... antagonizing them and increase the chances of them becoming what you fear.