There's actually a type of argument that could be used to say it's rape(not saying it is). Its that argument of wage slavery vs official slavery, an argument that's hardly addressed despite the fact that it needs to be. The jist of it is the notion that it is not necessarily more moral to coerce someone into terrible working conditions without alternatives that wouldn't threaten them physically to undertake than it is to outright own another person.
If you looked at a really vague summary of what happened, you might think it's something like this. That as a slave without the option to leave him, she's technically coerced into being with him. The thing is, that's not the reality of this specific situation, as they made it clear that any obligations she had in order to maintain herself and her livelihood did NOT include sex.
...the real problem is that he still officially owns her without any kind of justification as to why (like how Shield hero owns a village of people later on to both make them stronger from the game mechanics of the story and to make attacks against them considered attacks against himself who is one of the most powerful and influential figures). Not saying I care, cuz it's just a story, but if somethins gonna raise your eyebrows, it should be this. Thats what a normy, who hasn't been drip-feed anime tropes for years like we have, would worry about.