Cale seems like an interestingly flawed character.
Not evil or immoral, nor good and upright, certainly: But, crucially, remarkably irresponsible, contrary to the image he's cultivated—he was pretty much the sole person in a position to do something about vicious cycle between the Duchess and her maids, and he just sort of blew it off, it seems. As if there were somehow no way that could come around to cause trouble, like it's threatening to do right now...?
Like, even if we just want to say he's a callous meritocrat who only cares for the functioning of the house, he's fundamentally failed at that by just letting things fester.