@Irze
I mean, the man is super confused. No one but her knows she is from a different world, so to him, one day she woke up, had a complete 180 personality flip, and is now avoiding him and acting like even looking at him disgusts her (which I guess in a way it actually does). She is/was his wife, who he seemed at least to care about to an extent. She also "knew" what she was getting into by marring him. So her recent actions are defiantly a shock to him.
Remember that her entire world view is condensed to what she read in a book. She has yet to really grasp the full picture. He isn't actually the bad guy in this world. Sure his methods are extreme, but only by requirement to protect his family. IIRC, he literally is the only Noble family not corrupt, protecting the King. So all other families target his to remove him and his influence, so they can get at the King to protect their own self interests (basically greedy and power hungry nobles being greedy and power hungry). The book though, portrays the world, not from a political or an economical standpoint, but a romance, in which the protagonist is from one of the other families.
In other words, she has a flawed view of the world. And her actions are shocking for the Duke since it is a 180 from her prior version. Kinda feel bad for the guy a bit since by no real fault of his own (since she apparently was informed of what she was getting into before they were married), his wife suddenly wakes up one day and just fears and hates his existence.