I don't think he hates her but rather the memories she keeps alife that he wants to forget. Or maybe the part of her resembling part of him he wants to move on from.
But either way, other than knocking down his phone (which wasn't intentional either) she's never done anything to offend him so I don't think he literally hates her...