One theory about that scene that I've heard people talking about is that Takami's actually misremembering what was said in that conversation, because when you look back at the chapter when Ayame suggests that he break up with Nogi, her words and facial expression is different. It could be that Ayame actually said something else and Takami misinterpreted her. Also, I don't believe that Takami would say "I'd feel pretty bad for her"; it's clear he doesn't really care about people's feelings anyway.
There's a big problem with that theory, well more than one
There's cahpter 10 in which we see her feeling herself after stealing Takami from Nogi, looking down on everyone else, that's from her own POV and inner monologue
Then the entirety of chapter 19 in which Ayame is shown bad mouthing Rin and complaining abou everyone else, using the exact same rhetoric shown on the flashback of chapter 21
That is repeated on chapter 20, and is something Rin planned because she knew that Ayame has a bad personality and that with a bit of a push she would end up digging her own grave which is why she tried to push the image of wanting to go pro and have Ayame give her guitar lessons back on chapter 18
That's all culminating on chapter 21 when the plan goes into action and Ayame is finally dumped, and then in her own flashback we learn why she has such a bad personality
That's 4 chapters showing us how Ayame behaves
I don't think Takami conveniently misremembered a behavior that goes pretty much in line with the troublesome actions Ayame continuously displayed for 4 chapters, and that goes in line with how the only way she managed to make herself feel better was to look down upon others (explained on chapter 21), which is why she is the one that:
The one with memory problems is not Takami, is Ayame who deliberately forgets what she said because it worked on her favor