From what I remember from the novel, he doesn't want a harem, just any girl who would love him back, but misunderstandings or poor timing keep screwing things up. The harem part ends up forming without him even knowing, as all these women run into each other on their quest to track him down and explain things properly, but they keep being one step behind, arriving just after he leaves. Unfortunately I remember the novel translation dropping right around the time the formula started changing.
Though as much as I preferred the way the novel handled a lot of things, I feel like I remember the story taking some weird turns right around when the translation dropped. Been years since I read it, so I might be mixing it up with something else, but if not, maybe different is better for the manga.