It's glossed over a lot but her sister apparently drugged and raped him? And I think her dad beat him up once too IIRC.
Direct quote from novel translation chapter 25 out of 26:
"
"I know, I know. I know that my sister drugged Kohei with sleeping pills and attacked her. I thought my heart would stop when I saw the two of you naked in bed......."
I came to the house to have an important talk, but I lost consciousness, and the next thing I knew, Yuzu was there in the shape of a demon. And your sister said something dangerous like, "I wanted to ruin my hateful sister's happiness"
"
Which seems pretty big to only mention it one time near the end of an otherwise lighthearted novel. It seems mostly thrown in as there wasn't many other reasons given for their animosity and divorce.