Wait, I think I figured it out... he's playing on "Hard Mode" now, but everything gets reset each time he "conquers" a heroine and improves their situation..he's just setting things up for the protagonist's routes. Which means he has to watch as he gets NTR'd as the main protagonist's routes...
It probably wasn't her, but the cliche he protected her and that's why he lost his wings + she buried her memories of it.
I'm also betting on her looking different back then, which would be a reason he doesn't recognize her. But that's just speculation at this point.