Did we read the same story? Sure, the husband sleeps around, that much is confirmed.
But didn't support her in her grief? Didn't stay by her side? She locked herself away for months, completely shutting him out despite his efforts, and then came out swinging with the most insane shit in the world. What's he supposed to do? Support her in this delusion like Doubara?
How is he abusive? There were two scenes that could be construed as "abuse", and both were him attempting to make her face the reality that their child is dead, shock her back to her senses, admittedly in a crude way, but both times she again shut him out, ignored it, and forged ahead on her ridiculous path.
Doesn't help that their (supposedly) mutual friends focus on her exclusively while more or less ignoring him, everyone's talking shit about him either to his face or behind his back, his mother basically told him to "your child's dead, but you gotta nut up now", later following up with "maybe if you had another child..." (compare and contrast with Doubara's mother telling him that a parent losing their child is the most tragic thing), and the most support he ever got in his own grief is from his boss of all people, who told him to take time off work.
Meanwhile, don't forget that he also lost his son. Haruji is clearly showing signs of an intense depression, he drinks like a fish, smokes, doesn't take care of himself or his surroundings, distracts himself with women (admittedly a shitty thing to do), and lashes out, and everyone around him either doesn't give a shit, or blames him for his suicidal delusional wife running off on a mystery quest. All he ever got is pitiful whispers about him, but no support. No words of encouragement. And then his wife, the one person he expected to share in his grief and truly understand, comes with the cuckoo reincarnation stuff, that's when he lost it.
Reread the story sometime, and try to see the husband not as "the man who isn't the protagonist", but a human. And he might not be a paragon of morality or upstanding character, but he's not some cruel villain. He's just a guy who's expected to bear all the weight on his own, and indulge his wife in her delusions, just because he's "the man", and he breaks under it.