Kouhi and Mitsuke aren't even just his servants. They are basically his two testicles, since he doesn't seem to have much balls himself.
It's embarrassing (as a protagonist) how easily he talks about himself in a demeaning way and how he belittles himself.
Even though he doesn't care, he should be angry himself, and not leave that to his women when someone just insults him.
Also I think it's just idiotic beyond belief to let a snake into his tower, and even give the guy a position with authority, even though MC and his servants knew from the start that the fucker was already thinking of trying to steal his position (at some point in the future)..
The guy is rotten to the core, and that same influence was already spreading to his daughter.
I just dislike it so much when these japanese low-testo protagonists are basically saints themselves, when people say or do bad things to them. So rare to see a guy with actual balls, and them not taking shit from people.