You're mostly right. However, in this world some people have willingly sold themselves into slavery and agreed to go become a slave to be able to repay debts or for whatever reason. In such situations, they fully understand what it means to become a slave, which in turn means they are consenting to it. Furthermore, it's possible that some slaves want to have sex. They might have been forced into being slaves, but that doesn't mean they themselves don't have a sex drive or a desire for it. That, or they willingly chose to become slaves, and thus consent to the sex as well. But most likely by and large, yes it's probably slave rape. Although in MC's case, if I remember correctly I think Roxanne and Sherry sold themselves into slavery to help support their families. While they might not have been keen on all the aspects of slavery, they nonetheless still made a decision to become slaves for the money.
At the end of the day, if it makes you that uncomfortable, just don't read it.