Pretty sure he's not the worst man. That would be most of those princes who abandon their fiancées for the new girl, or those who kick out the MC from the party for being a burden. And while not men, Avril's family. The man there is just crap, but not as crap as the other two.
If you don't act like your father, I'm pretty sure that person won't feel the same way. Blood isn't quite that thick.
He gave his name as the Duke, but the problem was it was too expensive? Does he not think the Duke has enough money?