Heyo, the retainers are like other houses/groups/families within the territory but when "everyone here" is referred to, I think he means their family. At least, that's what I inferred from the raw.
It's odd. I don't know what in a raw but the scene and entire interaction gives different felling. Everyone having so much respect and love for her (as a war goddess) that Brest had to be strict with her. And with how muscle-headed they are no one cared about houses/groups/families (as we would see how they do theirs taxes). For me at least sentence should be like this:
There isn't a single retainer who does hate that insane woman.