I always found that odd too.
IRL example, Rome. Slaves were expensive so they were looked after as a symbol of status.
A filthy starving slave wouldn't make your status look good. You'd be seen as too poor to look after them.
Whereas you see in cultures where they were more affordable, treatment became worse.
You also have to take in account that the difference between free and slave is how much you one at birth and where you was born. In America the dividing line was race
There was also a time that slaves could buy their freedom, that ended in America also