That's the one problem I have in this manga.
He's always talking about having beautiful women to make a harem with yet also purposely push aside the beautiful women who genuinely care for him and would have no problem being part of his harem. Make it make sense.
I wouldn't even care if he ignored all the women in his life (except Rosetta, she deserves all the love) and just made his harem with Amagi and other automatas. Just stop talking about having a bunch of beautiful women when you're being...well, you.
He wants to be a villain but is even genuinely a good person, the trauma stops him by getting multiple womens around and his morality by just getting slave prostitutes, he wants a connection with the woman he will choose so he can't simply build a harem, nor truly wants to as Amagi satisfy his requirements of companion and loyalty.
His not acting as a villain is because the villain behaviour in his mind is equivalent to just a unique type of lord in that universe, he even doesn't want to get betrayed like a third class evil lord and therefore is making sure to build a good system under himself before descending into laviscis activities, that would still be super acceptable in there.
His request for bribes from a merchant was gold, that is valuable but like the lowest tier, just enough to say he is getting something.
When he raised taxes to show he was evil the people were fine with it as it was still in a reasonable margin.
