This chapter really shows how deep the problem his face was in that world. Thought to be a monster by almost everyone. His assholic action with barrier didn't really help to curb that. I wonder if these are bad moments with the tsundere, what were the really ugly ones he removed from his memory? Problem is, she isn't very good to him, uncle has valid points. She does have a good side, but very overshadowed by the bad. It's something you can appreciate in a normal world when you are at peace, where you encounter other people who put you in a positive mood and who are not hunting your skin.
In the world he was, there was no one to help or guide him, confide, only this girl shadowing, mocking and insulting him. That only got him angry so he either retaliate or run away. Of course, he didn't open to her or read between lines. It's also her fault for not realizing how devastating the effect on his psyche that world had. You know doing the same click over and over expecting different outcomes is just dumb. The problem is after some point, even if she just walk up to him, said 'I love you!' and kissed him, I suspect he would just chalk it up as a joke anyway.