A minor problem of this manga is that these god damned servants think it's ok to neglecte their duty or even worse doing bad thing intentionally just because they thought the person they serve are supposed to be bad guy. Like did they forget their position and think they're hired to be a judge or something? I thought the duke would do something about it since he yelled at the short haired maid in earlier chapter, but then they totally sideline the matter. In your average villainess manga the MC would have give them hell after acquire back their power.