Simple : the moment he choses an Harem over a Party, everything happening to them was on him.
Are you for real?
The only things that you reproach him is to be horny. Ceres and the mothers can't possibly make any remarks to the kids about that because they have spent more time fucking than the kids.
Out of all the "kicked out of the Hero's party" stories out there, Zect's party is among the least scummy people. They didn't backstabbed him and let him to die, he still has all the privileges of being a member of the Hero's party, they never blamed him saying that he cursed them and swore to get revenge,... They immediately understood than they need Ceres to take care of them and wanted to apologize and ask him to come back.
Those basic things that they don't know to do should have been tought to them by their parents, in particular their mothers btw. The mothers screwed up their kids education and now they are here to punish the kids for the poor education they gave to them...
Again, the first big loss of Zect and his party is the dragon stampede created by Ceres himself. Their big crime is that they didn't know how to organise the army of the city. But it is completely stupid to ask the leader of a four-people party to do it as he has obviously never received any training for that as other people know the terrain and the forces at their disposal better than Zect could.
And the second, they would have been defeated just as hard is Ceres was still in the party as the demon general is stronger than the demon lord himself. The Milfs can make fun of their kids if they want but they would have been humiliated just like their kids. The only reason Ceres could win against Mamon was because he has been transformed into a dragon god (again, as a reward for causing the dragon stampede!).
How is it on him?
The mothers should just thank their kids for having taken Ceres in their party and then have let him go. If he hadn't been a member of the Hero's party, he couldn't have a harem and mary all the milfs, and if they didn't let him go, the milfs would still be stucked in their unhappy marriage. And again, Ceres didn't mind cooking for the kids or Zect having "fun" with the three girls as he never liked any of them.
It's just the milfs who are taking their frustration on their kids. The kids have never done anything wrong to the mothers and Ceres didn't mind what they did.
But the milfs are only going to get more and more unfair to the kids as the story goes on. That's why I'm pissed at them.