With that out of the way, here's my analysis:
Hakari (the pink-haired deredere girl) and Karane (the blonde tsundere girl) have kissed each other more than once. This happened first in the "Kissing Zombie" mini arc (or whatever it's called), but repeats itself at least one more time (as far as I remember) in a later chapter, while they're asleep, much to the Rentarou's joy, as he views that as them getting along.
On top of that, the harem members are really closer to each other, and considering the Rentarou's views on relationships, he'd likely appreciate it if the girls started getting involved with each more intimately.
Aside from that, there's a more explicit example of the story's polyamory subtext: the newest harem member, Hakari's mother, Hahari. In one of the last few chapters, Hahari admits that she's still in love with her late husband, and the protagonist assures her that there's nothing wrong with that. Well, all right. You could argue that, while he isn't exactly comfortable with sharing her heart with someone else, he told her that he didn't mind it so she would stop beating herself up over it. However, shortly afterward, Rentarou meets the ghost of her late husband, who is revealed to have been watching over his family all along. He tells Rentarou that he hates him for taking both his wife and daughter, but Rentarou assures him that he loves them with all his heart. The ghost forgives him, and tries to pass on, but Rentarou stops him, insisting that he remains in this plane, instead of passing on, so he could love Hakari and Hahari together with him, sharing the mother-daughter pair. Hahari's husband still decides to pass on in the end, but Rentarou has still made his views on love quite clear. He doesn't mind sharing at all.
So ... yeah, that's pretty much it. When I said it's more like polyamory than polygamy in my earlier comment, I was talking about the attitude of the protagonist. He doesn't have an iota of possessiveness in him. He just wants the girls to be happy, regardless of what that might entail.
Now, as you mentioned, there are people who claim that this is a great harem manga. In a way, it is. But it can't be denied that there's a polyamory subtext. This isn't a standard harem (though I may be biased, since I'm more fond of the stories where the girls bicker and fight each other due to their jealousy;; I've always been fond of that sort of drama). However, it's important to note that, lately, in the realms of manga and fanfiction, there has been a surge of stories that sell themselves as harem stories, even though their protagonists are clearly in a polyamorous relationship. Therefore, I wouldn't be surprised if this story was still called a harem story, even if the polyamory was more evident (most people don't know the difference to begin with), since that's apparently the trend as of late.
Wrapping up, keep in mind that this is all my analysis, and that I may be reading too much into it. This is, after all, a gag manga. I don't think the author is too concerned with whether it's about polyamory or polygamy.