Look, his actions are very scummy, but consider the following:
1) He isn't that smart.
2) His continued living does actually hinge on him having sex with an actual woman (or at least not whatever the maids count as).
3) He has been lowkey brainwashed into listening to the no consent fairy, as his only real companion.
4) We don't know if he'd actually do more than kiss or grope before his very loose morals might have hung up.
So even if he wasn't typically the type (doubtful, I know) to have sleep sex without his life being on the line, we'll never know.
All that said, rape is still rape. Even if you can basically view it currently as a crime caused by duress, most places don't count sexually violent crimes as defensible by duress.
That also being said, I'm not here to debate the moral quandary of the protagonist in an erotic manga, but enjoy the ride.