I just don't quite see it that way, I started the LN once but droped it due to manga. IMO his downfall was Aika and not the Family. Even if family was involved it had a very minor impact as he stated to his sister at the rooftop. He dont have "low self confidence" he just got aware that he's not some ikemen like student council President. Realistic and low self-esteem are different. You got confused like his sister did. He said it himself, he know himself and that he's complitely "average" that's why the title is "The Dreaming Boy is a REALIST"
He WAS quite remarkable in terms of Study ect because he put lot's of effort, you can see it when his grades droped when he stoped to tryhard.
Family is not to blame, again, he said it himself. Aika was giving him NO for about 3 years now so he just assumed that he is "not good enough for HER". If you still remember he even asked if she could introduce him to girls from "his league" which is nonsence for "low self-esteem" person. He got REAL not LOW, Boy is a REALIST