@tseng i think you got the wrong picture to explain the situation in the next chapter.
First, no, he's not being pushed away by the girls, that's just all in his mind, he is afraid that he can't hang out together again because now Shikimori become popular among the girls.
Second, also no, the...