It was stated flat-out in the bonus chapter of Volume 6. The unfortunate part is that the official translation completely misunderstood what was being said and dumbed it down to "repeated test failures" when him failing his entrance exams was likely the final nail in the coffin for his self-confidence.
I've been meaning to get around to making a proper TL for that chapter since the earlier fan translators for the series never covered it and the official one missed something so huge.
It's not surprising Mahiro failed his college entrance exams, because he's shown to be a very slow learner about a lot of things (going to the bathroom often enough being the biggest, but we can mainly blame that one on Nekotofu's fetish). My guess would be that his self-confidence was so low by that point that he
expected to fail and it became a self-fulfilling prophecy.
It's a shame the official translations screwed that up so badly. Definitely doesn't make me want to read them, much less buy them. Not that I can afford to buy manga nowadays, nor am I willing to buy digital manga at all*, plus I don't want physical books any longer (I don't have the space for more of them), so it doesn't really matter that much.
* Any publisher upset about that has JManga to blame for fucking up so badly and taking away the manga I paid for. IIRC Crunchyroll was involved in that, so they still have someone still in business to yell at over it.