It's weird how the author went out of his way to vilify Zect early in the story, only now to turn around and make him appear sympathetic. Why spend all that time making him the bad guy, and making him look like a complete douche, if he was going to just apply the power of friendship and make everything all better? Seems like poor writing.
Except, the author never really did that. We just fell in the pitfall of comparing the dude with similar scenarios we see in different isekai stories.
Even the MC was very adamant about "not wanting to punish him", and just went along to please the mothers.
If you pay attention, even when he was "punishing" their ex-husbands, he made sure to do it in a way that would better them as people.
Though one of them - WE KNOW EXACTLY WHO - should've just been left to die. What he did was NOT forgivable at all. And I really wish Japan would start to lean more that way with this topic, instead of just "oh no, you're such a piece of shit that don't deserve to live, and you showed that over and over again by doing this very specific disgusting thing, and you crossed 50 lines while you were at it... but here, I'll give you another chance!" - then, same MC proceeds to kill someone who didn't do half of that, for no reason
coff dragons coff.
That aside, though, Zect wasn't really written to be the worst - other than someone who tried to NTR the protagonist... which is probably not even what happened, given what we saw everyone, including the MC and his "lover" say later.
In fact, we've just let ourselves be baited by this, even though the author made sure we knew Zect and the party weren't total scumbags like in the other stories.
You can even see that in the very first chapter, when the guy is like "So you're leaving the party?", and they also say things like "You'd probably end up dying if you come with us".
So in a way, they were trying to protect him, even if they're all scumbags in a way or another.
Either way, they don't compare in any way, shape or form to the other "banishing parties" you'll see in other isekais. Which is completely astonishing to me, as it came from this barely-not-hentai work.