I want to cut him some slack and I really want to understand the conundrum he found himself into, but at the end of the day it was all lack of balls, he both didn't do anything wrong and he did everything wrong.
This is my best feelings on the matter. Sure, going over to her place whilst people were watching may have been risky, but he could've at least waited until the low-life paparazzi gave up stalking her premises, whilst texting or somehow communicating to her that he was going to come over, it's just that if he did at that very moment, things would just become worse for her with the public.
He lost a damn good woman, and I'm not just saying this because she's a hot, busty idol. She was a genuine, authentic, down-to-earth girl.
Speaking of, why the hell is this jailbait Ayano the main FMC of this series? It should've been Akari, their backstory has enough potential to stand on its own as its own manga.