I guess even though the duke wasn't directly involved, his servant, who is an extension/representative of his noble house, caused enough damage for his employer to deal with the consequences. Hopefully seeing all this and having to give up his daughter to become a maid has opened his eyes enough to realize how his neglect on her discipline was partially to blame on all this.