If I understood Ichikawa correctly, he meant that he couldn't handle hanging out with the basketball club. Too many people, he doesn't know them well ⇒ he doesn't want to repeat this experience ⇒ all the future hang-outs will be with people he knows, i.e. with Yamada.