Other than the second wife, I don't think his immediate family needs much comeuppance. His brother is only 4. We haven't seen the father do much that is terrible (and he is already getting some from his first wife and child abandoning him). The worst that we saw was his mother being sent to do a servant's task by his grandmother. While not done by him, it is something that should never be done to a wife, and suggests some systemic problems that should have been addressed by him.
The precipitating event here doesn't appear that bad. They did come in two carriages, so it wasn't like they were left behind without a vehicle. Heading to an afterparty with the prospective family of the child that just got engaged & focusing on that child on the engagement day is completely appropriate. The father clearly told the archduke's servants why they were leaving early, an oddity that makes it appear he was trying to relay the message to these two as he likely did not know where they were. If anything, this appears to be a gaffe on part of the archduke's servants. (Admittedly, I don't think that is the author's intent.)
That said, at the point where the wife and child's response to the question of 'Hey, should we leave him?' is 'It's good to hear you feel the same as me. Let's go.', the dude has screwed up royally.