Ah, so the dad disappeared because of an illogical reason. I guess it was either what I said back during the flashback, or the husband being an unreasonably huge POS.
They went with the second. And then the mother, who insisted on having Sayu be born, blames Sayu for her husband being a POS...