@Kynnath Exactly this. The duke's character makes no sense. I had to make sure I got the timeline right. It would be a little more believable if they lived in separate households (minimal contact) and/or he was away at war for the majority of their engagement. I feel like the author didn't think this through.
Also, I'm a little confused as to why they were engaged in the first place. Usually this cliche is as follows:
guy and girl are in a "pre-natal engagement" (so I assume, decided before they were born/conceived) arranged by their aristocratic families to have familial ties with one another (because they're good friends). It seems to me her family has abandoned her, and the duke's parent also don't like her. Why did the engagement happen when their son was so against it?