I think it would be the opposite. It may make the Duke think she was going to do something that would force him to "love" her, like brainwashing or magic.
The line where he asked what will happen if he'll end up falling in love with her really caught me off guard. That was very unexpected and probably the first time I read something along that line.
Also, wtf duke's clone hahaha
I’m guessing Clea Prime got him because he has albinism too. In which case it’s really weird to show up at someone’s home with a slave that looks like them. 🙃