Dude definitely doesn't trust them and is on high alert. He didn't even take his coat off indoors, so that he can flee at a moment's notice when needed.
I bet there were even teachers who singled him out because he's the son of a murderer. But apparently he wasn't bullied mercilessly, unless he just lied about it to avoid Sakuya begging for more details.
Btw, nobody forgets painful and scary memories. Maybe you don't think about them all the time, but they aren't forgotten, until dementia possibly claims you. That's the sort of thing memory was only used for initially when animals started developing. To be able to avoid the known dangerous things in the future. Of course the other side is remembering nutrient sources and matters related to mating.