Historically, not ever slave was treated horribly. This was commonplace in the US but farther back into even the antiquity, slaves were extremely valuable and treated with care. Working adults didn't grow from trees after all and only the super rich heavily abused them like you'd see with most cartoonishly evil manga villains. In fact, a lot of people advocating for slavery used the argument that "slavery isn't all bad because we don't all treat them horribly" for thousands of years.
The problem with slavery isn't """just""" the fact that some slaves get abused, it's that the entire system undercuts human rights and turns people into objects. This is magnitudes worse in isekai's like this, where it isn't a longstanding abusive system of dehumanization but a full blown magic which deprives a human being of all of their rights effective immediately. There is no "out". All of Naofumi's slaves are indentured to Naofumi for all eternity unless Naofumi let's them free.
The argument that Naofumi is justified in having slaves because of trust issues and because he doesn't hurt them IS glorifying slavery. It's an unjustifiable offence and even saying "but in some cases it's okay" is a slippery slope in promoting something. Raphtalia's case was a best case scenario, she was near as hell dead and Naofumi was at his lowest mentally, even then, she wouldn't have been there if slavery wasn't a thing to begin with. The other kids Naofumi takes in are much worse, because they became slaves as a direct consequence of Naofumi taking in Raphtalia and skyrocketing the value of beast children. And it's a literal plot point that Naofumi regrettably tries his best to take in as many as he can to "save them". Furthermore, the argument that he has trust issues at this point in the story is like, irrelevant? They are children, what are they gonna do to Naofumi? Bite him in the hand? And he could've still been a protective patron by telling others he made them slave but not actually doing that, lying is an option here.
Last but not least. This would be okay, if Naofumi was showed to be in the wrong at any given point in the story or if he ultimately tried to abolishing slavery in the long-term by crashing the market or some shit. But that is the farthest from the case. He is praised for "saving the children and giving them a new life" and is literal friends with a slave trader. You cannot get closer to portraying slavery in a positive light if you tried.