What even happened to him back in his childhood? What was all this about?
I think I'm sort of following it, but this whole story is like... trying to be deep, but failing at it. It's like a not-smart person trying to write a really smart story. It reminds me of OreGairu but without the intelligence required of the author.
I think the basic idea is that because of what happened with Shiori (people around them becoming more aware of the difference between boys and girls, maturing, becoming teenagers, that sort of thing), and that being pushed onto Jirou and Shiori, Jirou basically couldn't handle it, didn't know how to deal with it, so instead rejected everything and isolated himself out of fear of being... judged, made fun of, idk.