Ignoring the bits when he was a child, literally all of the other instances of people "being mean to him" are justifiable. Like, you literally tried to kill the doctor while being racist as all hell, of course big sis and the doc are going to say rude things to you... Also, you were just told that Kaia can be abrasive to people, so why are you lumping her in with the rest??
I like this story, but the author is trying too hard to make us feel bad for this dude by telling us we should feel bad... Except they're using scenes that just remind us that he's an ass. Doesn't help that before this we were straight up told that he's here to spy on the Sun Witch...