Its kind of funny to me how many are cheering at him for not buying a slave and mentioning how morally disgusting it is, but have failed to realize that the thing that makes it morally disgusting is how they are treated. Many of the MCs don't treat them as slaves but as a companion and a person, far closer to a modern employee than as a slave. On top of that him not buying one doesn't free them, they will just be bought by someone else. And if bought but someone from that world they are far more likely to be treated as actual slaves. A argument can be made of not supporting a industry, but one less buyer will not make any difference, if he talked to other and did anything to shift common morals that would be a difference. But short of starting a revolution they won't help any of the slaves, but at least by buy one and treating them well you have saved them from most likely a far worse fate.
While I agree with the overall concept that judging things with nuance is better than making generalizations, I do think there are some things inherent to slavery most people can dislike. The big one is that the slave cannot choose to leave or do something else. It's similar to how people view capitalism as superior to feudalism because capitalism allows social mobility, even though
workers under capitalism work longer hours and have worse conditions than the average under feudalism (Judith Schor, MIT). Individuals can make a value determination on whether they'd trade social mobility for fewer hours worked.
As this is an isekai, the external values of their alternative to slavery must be considered. We don't have a lot of knowledge other than the slave sellers seem shady. For instance we haven't seen a horrifying war that rounded up their war slaves. The Isekai version of Debt, War, and Criminal Slaves is closer to Roman Slavery than American Commodity slavery. Roman Slavery was awful; debt slaves and war slaves didn't deserve their fates, and some crimes didn't deserve the sentence of slavery. It makes me inclined to think that overall, isekai Slavery is still evil.
The further question is what can the MC do. Not buying slaves is a kind of minor boycott, it passes a purity test but doesn't effect any changes. Similarly buying a few slaves and treating them well is a minor boon to a few individuals without broader improvements. He could kill the slaver and free a bunch of slaves, or start killing multiple slavers. Or he could get OP and destroy the entire society and build a communist utopia (When are we going to get Chairman Mao Isekai?).
Moral discussion of slavery aside there's a different judgment to be made about the story. Isekai Slave harems are a huge cliché that borders on rape fetish, and at least this MC didn't instantly do that.