I agree your critic, that bothers me too.
The flashback and ending did show that he is fond of her, but the side where he feels bothered is also real and communicating it is important.
I don't know how to write/show that complex "I like her presence" and "she bothers me" feeling better though.