Anyone who's complaining about his apathy and how he supposedly acts counter to the initial point of the story, just read this and let it end there.
You should probably read between the lines more instead of taking what he says at face value. The reason that people were able to use him to approach his brother in the first place was because of his personality. He's too trusting of people, too much of a nice guy, he let's people in really easily. (and also not trusting women isn't the same thing as holding grudges against them, he doesn't actually hold grudges against people cause he's too nice for that)
The thing is, he knows this. He realizes that people take advantage of his friendliness so he now attempts to put up walls to protect himself.
The issue is he's still a really nice person, he can't change his personality; so if someone's got a problem and they ask for help, HE CAN'T bring himself to refuse because that's just how he is.
Those walls he puts up act counter to his very being, so they have flaws, holes. They aren't very hard to break down if you don't strike where he expects you to. She fessed up and apologized, which is something that his "walls" aren't equipped to defend against. Same with houjou, her actions are nothing like the girls that use him, so she's confusing.
Tldr, all that matters is that he's just way too nice to hold a grudge or stay mad. Especially because he's gone through it so many times already. You might think that's dumb, because, yeah, it kind of is. He should be mad; but it doesn't act against his character. It fits it perfectly.