I know he doesn't want to have anything to do with her, but wow. He really ignore everything , like her words aren't even entering his ears, he isn't even looking at her and he really doesn't care about her even a bit. Even faces with her whatever feeling she had, his mind is filled only with "what a pain" kind of thoughts.
It is understandable stance, but ngl. If I'm in her shoes, it hurtful.
I think, his refusal confront her feeling and dismissing her whole being is just receipt for making bigger trouble.
Just wanted to chat/discuss (no hard feelings).
Actually, he really cares for her, even if he doesn't admit it as much as he does. He even prepares for a shoot if Christmas with her mom gets cancelled. There are countless times where he spent time with her for "work," even though it is not needed. Yeah, he might have those feelings of annoyance, but his worry for her overwhelms that. After all, he's not a "personal" type of guy; he focuses more on his work because it is the only thing he's good at, according to him.
The main issue with their relationship per se is the lack of definition between their relationship and the fact that they can't solve it that easily. Currently, Fumi sees himself as just a manager/guardian taking care of Ayano. Fumi's time spent with Ayano is merely seen as "required work" rather than quality time with her. They both know that their time spent with each other is "not normal" for work, but they don't want to point it out because it would destroy their relationship.
Ayano can't define it because she is scared that Fumi will distance himself from her, which is why she is so jealous of Hoshi freely declaring her love to Fumi. She is scared that once his "work" is done with her, their relationship will disappear (which is why she hates Fumi describing their time spent as work).
Fumi, on the other hand, can't define it because he is unsure of her feelings for him, and frankly, he isn't able to. As the other managers pointed out, you can't trust the feelings of a middle-schooler because you don't know if they understand it themselves. On top of that, it is an open secret in their workplace that both of them are weirdly close. That is why he always gives simple questions/responses to her, trying to not mix work and personal life. The problem is the moment he acknowledges it, it is the moment he has to choose, and neither side will be good to him.
Which is why, as cruel as it sounds, ignoring it is the "safest" option. It doesn't open the can of worms about their relationship, but it leaves their relationship in the grey area, making their relationship blurry.
Now don't get me wrong, I agree with your sentiment that letting it boil over will not be the best case scenario for both of them, and defining their relationship would probably be best for them. But with the situation they are in right now, there is no easy answer. Which is why the most pivotal part of the manga is this: to realize how important the other person is to them.