To be fair, the main character is from modern Japan, so they would have those values.
As for the other characters, while it has been part of human history since the beginning, it was often seen as more of a "necessary evil" and lumped together with other morally dubious businesses like prostitution. They were usually either criminals or prisoners of war, after all, and so many wouldn't want anything to do with them.
Also, in most areas where it was used, they were usually too expensive for any but the wealthy to afford, so it would probably have been abnormal for the average person to encounter as well.