Sorry, this post is a bit longer but I just had to get this off my chest.
The problem is that the very core of the story doesn't work:
Athena tries to "save" him by treating him like scum and (surprise surprise) he dives into a dungeon and nearly dies.
If you're trying to focus on the party who exiled the op party member (I think the main theme will be cheat ability vs effort) you shouldn't make them friends or let them seperate in a friendly way.
The way it happened in this manga makes it impossible to like any of the party members.
Athena: In the way she parted with him she should realize that she left him with nothing. He lost his place to live (literally), his party members/friends and the love of his life (even if she doesn't know, she's still his best friend/childhood friend). There are two possible outcomes. Either he's total devastated or he starts to hate the people who treated him that way (I think he chose both after he even tried to reconcile and the "misunderstanding" happened). The author tried to make her the damsel in distress saved by the NTR-MC (saves her when she nearly dies, supports her, defend her in front of Est), but though she's Est's former best friend, she doesn't notice how broken he is and doesn't question what caused the change in his behaviour. Making Est the villain simply doesn't work!
Daru: Though he's supposed the MC who saves the day, he is the most detestable character in the story. In everything he does he seems to have ulterior motives. He is the one who knows all the truths and could solve all the "misunderstandings" but he doesn't. He knows that Est loves Athena and thus why Est is so angry but he doesn't tell Athena what happened that night (he's not "drowned in power"; nice try author). I think he's even intentionally provoking Est when he takes Athena in his arm in this chapter thinking about what happened that night. He treats Ets like a stranger and seems to be happy that he got rid of him ("Was this girl always such a cutie"). I think it's really hard to find someone who likes the thought that this "good" guy could actually suceed in winning Athena (except NTR fans).
The other ones are just lame side characters (especially the guileful elf).
Against the intention of the author Est is the only person you can sympathize with. For the rest of the party (who betrayed him and tried to make him the bad guy) you just wish the worst. Thus, the bad ratings are not really surprising because it's not fun to read a story about people you just have to hate and pretend they are heroes.
PS: Does anybody know how the story between the three of them ends (webnovel). Tried to read it but was not masochistic enough to finish it.