You know, Ashe and Jean also kinda look alike. Either way their kids are going to be totes lady killers.
He would never have more than a single child due to his trauma. If it turns out otherwise, it means the author was making much ado about nothing, which would be exceedingly strange considering how this whole series started and how it was the basis for the MC's personality. Just about how hearing that single argument between a brother and sister at he adventurers' guild resulted in Jean never going back there, instead selling his loot to the merchants' guild.
Though all that being said, this isn't exactly a high quality story, so the author abandoning the whole idea is also possible. In fact Jean seems to be enjoying watching the three kids of the farmer family, as if he had never had the bad childhood himself thanks to a sociopath sister. When Tina, the daughter, declared she will marry Jean, it also didn't seem to trigger his trauma, even though he had been subjected to repeated joke confessions arranged by his sister. So, maybe it's already happening that the author wants to forget the whole thing.