Because the way he behaved differently was in that he was acting more awkward around her, where if changes came from breaking up, it's more likely he'd be depressed.
I just assume Aya had told him everything via text in chapter 18 since he was on his phone while he was being awkward. Would explain why he was acting a bit off in chapter 19, 20, and 21. And why he dipped out so quickly in chapter 20.
However, if the scenario is that due to the "confession" at the fair he realized he still had feelings for Hikari and broke up with Yami, which might also be why he wanted to avoid her school. And the reason for the slap-then-kiss. Emotionally confused teenager
That doesn't make sense. The entire "confession" at the fair only happened because the dumbass friend at chapter 16 outed Yuu in wanting to be in the same school since he told Hikari the school he's attending was his number one choice.
Bro literally had no choice but to explain to Hikari his reasoning because at the time, he still thought and believed Hikari never had any romantic feelings for him and thought it might ruin the vibes and their friendship if she thought he loved her or something. So why would he break up with his GF that loves him, and he loves her, for a childhood friend that at the time, he didn't know loves him. He had everything to lose and nothing to gain from that.
Also to further prove my point, it literally takes Hikari being taken to a secluded spot in the fair on what might be a confession on her part to finally get the idea that Yuu might have liked her.
Idk, seems too much High-risk for almost no reward if you ask me.