Bros got a harem and keeps pissing them off becsuse he cant sit still
In actuality, he doesn't, and can't form relationship (like being in a party or marry someone), because of his "Loner" skill.
The only way he can have partners is by making them his subordinates (aka., slaves) with his "Servitude" skill, as at that point they would be treated as personal property rather than individual.
So, he doesn't really care what they think about him, as MC's end goal is to find a way to send all the girls back to Japan (in case of the boy's, he left the decision on whether they want to leave or not to them).