That depends on the websites what they want the tags are referring to.
Like I mentioned before, some websites even put lolicon for this series.
MC is not lolicon, maybe some readers are (for their website).
So it's neither right nor wrong since there is no official guideline; only from the fans...
For those who wondered why this is under Isekai Tag, I'll share my 50 cents:
異世界 - (Isekai) = 異 (Different/Strange) 世 (Society/Generation) 界 (World/Boundary)
By dictionary it can be referring to different world, parallel universe, alternative world.
If you on Earth and doing a documentary...
Yup, that is what I mean by assumption of this culture.
Plus mangadex definition is taken from Anime-Planet with no reliable references.
According to wikipedia, the word harem is taken from Arabic; something like a king with many women.
And by dictionary it means one man associate with many...
1 man + 3 girls + 1 female adventurer = single person + many opposite sex = Harem.
It is the right tag but most people have the wrong assumption due to how "harem" been associated in this culture.