Misundestranding the manhwa!
I don't understrand how can't the duke or his butler see the problem, sure they heared other talk about serena. Also the butler seems like a wise man why don't he undestrand that having a woman not related whatever so near an engaged man isn't good news ?
I hope now that we have a potential vilainess (the girls in the last panels) maybe we will have some plot progress.