This is more or less bad writing...what is the point of having an MC whose whole shtick was being overpowered due to the fact that he had the MEMORIES of being a sage in his past like, being weaker than anyone else and have none of his memories as a sage?
It would be one thing if it was only his power than was taken, then we could have seen how he copes with the loss of his power and show us how he manages to compensate only through knowledge.
But adding memory loss at that, is just stupid. What is the mangaka thinking this achieves?
"Oh, look!, I put the now powerless guy with complete loss of all his memory in a dangerous situation that he could normally clear with extreme ease now bungling it all up, because reasons!"
And no, showing us that the girls also developed is no real excuse...