I don't think it's that unbelievable if you remember the chapters with Azuha. They get along, after all. They share hobbies and he's trying to reach out to her. To her, he's a cute nerd boy who's bashful but trying to push past his insecurity. It's a totally reasonable attraction when you remember that the way we see their dates has been fast-forwarding through all the good times they have talking and doing things and then focus on his intense anxiety and self-doubt. There's no reason to assume he's particularly ugly or anything, so a normal guy with a stable job who you get along with and is shy and bashful around you? Just reverse it and consider what you'd think.
For Ametella here, she found his weird and embarrassing LN that literally only she is a fan of, and has decided that makes her special. She's no doubt held on to these feelings for years, without ever really considering their rational basis or exploring where they actually come from. Meeting him was bad for them both, because now she has a chance to make something that should've slowly retreated into an old fantasy happen and he's dealing with a choice that he is emotionally unprepared to make, being too afraid to hurt Ametella and not confident enough to believe he should actually pursue Azuha.
We are watching a man fumble a truly magnificent pass in this manga.
