I 100% agree, but it makes a twisted kind of sense when you think about it. Admittedly, I don't remember the exact details of how they all fell for him, but I'm pretty sure it was largely because he swooped onto the scene and positively impacted their lives in a big way (I think, by saving them, was how it happened).
From their point of view (and because the author clearly sucks at writing romance and knows it), that probably amounts to him riding in on a white horse and seducing them, so when they made their feelings known and he didn't reject them, it was like he was proactively trying to wife them all.
Plus, they're preteens in love and he's a literal demigod and apostle to their religious pantheon, so they probably think women fall for him on sight or something.
I'm not saying it's good reasoning, just that you can kinda see the train of thought they're on.