To be fair... it's not done well in many other Villainess manhwa from what I've seen....
I think the only one that did it well was 'My Dear Tyrant' (I think that's the name)
I read it as him not used to kindness in general, and that's what makes him uncomfortable; not MC being kind in particular, or her previous abuse of him.