@Doomer
Isn't it more like the possibles in Japan (club president and club kouhai), he couldn't believe they were actually interested in him and Mio & Tomoe he looks at like family (ie no incest), and so far he hasn't met anyone else in this world that he felt like that with and when he did try to visit prostitutes, Mio & Tomoe stopped him.
As for the ones that appear from here on forwards in the novel versions, i'm not uptodate, but the ones I might consider plausible usually have some way for him think of another sorts of relationship, instead of going to romance (eg teacher etc.)
So I wouldn't necessarily say he's scared of pussy, I would say he values his current relationships way more than the off-chance that they might actually get into a relationship and is afraid of the change in how they would interact in the group as a whole, depending on whether it succeeds or fails.