I'm not annoyed he didn't interpret it as affection, since he had every reason not to.
What annoys me is that he doesn't seem to have ever turned around and told her to shut the fuck up and leave. He just runs away like he did here. At no point does he seem to have said "people already act like I'm an orc wherever I go, I don't need someone following me around rubbing it in for extra cuntiness too. If it's that bad, leave me alone and don't come back". She needs some sense knocked into her whether you reject her affections, would consider accepting them, or don't even notice them.