@StaK
That's a bit silly. He didn't call her out on that because he wasn't that kind of person; even now he is someone who is focused on himself and who doesn't really think about how others interact with him. Moreover, it takes experience or being specifically taught about it to understand why defining these interactions from the start is important. It's easy to see that Kouhai has gotten into the habit of bullying him and clearly this is because he allowed it, but it's important to realize that he had no reason to understand how bullying starts or why it occurs. He's huge, strong with a martial arts background and looks scary; most people aren't going to mess with him to start with. In fact, most people were too scared to interact with him to start with, so he wouldn't have much understanding of relationships (for instance he has few friends) as he would always be the one initiating them for his own needs (say he needs to ask someone to coach him in swimming). Those relationships would always be very simple - he asks for something nicely and honestly and others would respond in kind - and it appears that in most cases it never went beyond that. Even now, he clearly interacts with many, but he doesn't put any effort into developing strong bonds with anyone (even though he's reportedly popular enough). Honestly, if Uzaki didn't forcefully encrust herself into his space thanks to her personality, there wouldn't be any relationship to speak of. He basically only has friends because others have decided they want to spend time with him not because he wants to spend time with someone else.
So really, he never had a chance to define anything back then and when we look at him now, what I see doesn't instil any confidence that he's changed all that much. He still doesn't have a clue how relationships work and he's only now starting to think about it. If Uzaki wasn't there to force him to think about it, he might never even figure anything out or only do so much later.
Of course, it's not like he's the only responsible for the nature of their interactions. Plenty have gotten closer to him without interacting with him the way Uzaki does. Uzaki is very childish in many aspects and while she's not a bad person by any means she's undoubtedly "difficult".