Food for thought: If he's going to be treated like a menace when he's done absolutely nothing wrong, then maybe he should just be a menace while finding some happiness.
Take the harem tag off. There's nothing romantic going on between him and the 2 other girls, unless there's something in the web novel I don't know about.