What I believe is pretty simple, Daru is at no point a bad person, he's just poorly written, as is everything else.
Don't forget that a lot of people here created an entire narrative of how he's some evil manipulative scumbag who wanted to ntr Est, that's what I always argued against, sure it was funny for a while, but it got old fast.
I think the biggest problem all the readers have with Daru is that he is not good or bad but more like a ... gap filler like all the other characters in this story. They are supposed to be good people but still have the function of the evil banishing party. As you already mentioned, at no point in the story Daru shows evil intentions. But on the other hand, his actions speak a different language: lovey -dovey-scenes with Athena, doing nothing to clear up the misunderstanding, hiding his powers when it't not for the sake of female party members, kicking out Est and getting another female party member => harem party.
Even though he is not supposed the "bad hero" he fulfills the role with his actions (which is not his fault but the author's as he's written like that).
This can also be seen as he is often confused as the leader (you did the same in previous comments). Actually, Athen is supposed to be the leader. With Daru being the MC and the leader at this point, Est's banishment makes even less sense. Nobody of his team members becomes stronger. Daru has to come to save the day and Athena still has no idea what happened during the whole story (misunderstanding, Est's feelings for her, smurfing party members) and is still weak. The only one who became stronger is the banished person (Est) like in all the other banished-stories, which wasn't the intention of the banishing-party's leader (Daru).
To put it in a nutshell, the problem with the story is, that it's exactly the same like all the other banishing stories except that the banishing party has "good intentions" and instead of actual cuckolding everything is a misunderstanding. The outcome is still the same (it's also hinted that he has a chance with two girls after banishing . The intentions and the actions of the characters are contradictory (their actions aren't good), thus the readers have to fill the gaps. Regarding Daru's actions, despite his intentions, he still fills in the role of the bad party leader.
PS: I only read the first chapters of the webnovel (couldn't take any more) but in the misunderstanding scene Daru appears even more indifferent: "It's Est fault because he ran away, he should have punched me." Also in the following chapters there's no compassion for Est. So I guess he's supposed to be that kind of character.