I think no harem is definitely a positive. And, with the flow of this story, I don't see how they could throw a harem at it. Not without ruining the story.
He has a harem of bros. Weapon shop bro, adventurer bro, and now we know guild master, body guard, and nobleman are on friendly terms with him. Maybe before he gets married he's gonna be surrounded by dudes looking for a meal and drink.