The perception of slavery as bad is technically not solely a modern era invention. Whether or not slavery is considered bad is purely based on societal and cultural norms. Even in ancient times, there were cultures that did not practice slavery. Also, even today slaves exist in the world, so thinking that slavery being bad is only modern common sense, is being a bit naive.
So please don't use your sheltered modern common sense to justify that an entire Era would consider slavery to be ok.
P.S That might come across a little hostile, but after studying a lot about history and a lot about slavery, I find many people's perceptions of such things are outright wrong or colored by politics.
All these points are rendered moot because, in the end, this is a FANTASY and any association with real-life events and or people are products of the author's IMAGINATION. So in this case slavery being accepted or present is just some fantasy of one Japanese dude.