I think the thing is that being pseudo-ostracized was actually, well, triggering, in the clinical sense. Something (besides just being compared to an overachieving prodigy sister) turned him into the hikikomori we see at the start of the series — it wouldn't surprise me if his otaku status or something similar got him excluded from whatever school society he was in.
Being excluded by the girls was okay when he could justify it, but once Asahi was all "Stay away! I promised not to tell you anything!" his pessimism won.