yeah, that is also another way to make the plot smoother. I could give it benefir of doubt and say the lucky pervert moment scene is simply fan service but then the inner monolouge(is she this cute all along) insta ruins it all.Better leave out love story completely.
Best thing would have been to put banishment before story with just a short flashback scene at the beginning: Athena remembers her childhood friend (best without a face or name; there was absolutely no need for the story to make Est part of it because they behave as if he never existed in the first place) who she banished from the party for his own sake. She convinced him (in a friendly way) and he returned to their hometown. Nevertheless she misses him and sees that shes's lacking without him.
Now you have just to add an elf girl who's not just a hardcore lesbian yandere but has actual character traits and a Daru who behaves like a teacher and not like a harem king and you would have a readable story with likable characters.
Comparedd to kaiser with his adopted daughters , the author set MC as always seeing them as daughters and nothing else even the daughter set an eyes on him so it didn't break his "father" character trait even we have fan service scene almost every chapter.
or if author really want romantic development between heroine and mentor then they shouldn't set heroine and banished in the first place. cuz it's just make reader can't root for them after they banish him.
there're several way to make it works but the author failed to pull it off. which is regretful since the premier of banisher PoV is interesting.