tbh, his group of "slave"/bodyguard is way more comfy to be around than 99.9% of "ore no slave harem" out there.
Well yeah, as much as I like pretty girls, if it's like real life I would rather surround myself with friendly or bro-like male or bodyguard too.
It's way more stressful to be around girls all the time, even if they have feeling for you.
You need to be aware about their feelings and pander/spoils them occasionally if you want to keep the relationship good.
You might even have to pretend or hide some aspect of yourself that they might not like.
And if you have any complex about yourself, good luck.