It can be more one-sided than they're all malevolently using him. Kaya's ability as we have seen it tells who that person's list of friends, but that doesn't mean it has to be a two-way street.
Hijiri a few chapters earlier went out of his way to put pressure on Sasamori regarding the burden she puts on Toru by confessing to him and called her unqualified to think she can understand him. When Tokita called that out as unnecessary, Hijiri's retort was that while they all are a group, they're only there for specific people and their collective relationships aren't mutual- ex. Tokita is only with them because Kuga is there and will always side with her over the others push come to shove. He ended his point with declaring he's Toru's friend, not Sasamori's, and everyone's going to end up realizing they're all not really friends with each other eventually anyway.
Kaya's powers here just expands on something Hijiri implied a few chapters ago to an even harsher degree than he thought- Toru doesn't consider them to be his friends, Hijiri himself also excluded from that list. He believes he is, but Toru in his heart disagrees and to avoid trouble has allowed that delusion to continue. It's likely the same for the others- they think they've become friends with him but don't realize he won't accept it. Knowing her brother has no friends, from his perspective, is why she concludes they're nice to him and spend time with him was because he found a way to make himself useful to them. Kaya's seen her brother be bullied and loathed her entire life, for reasons she confesses here she can't really understand, so it makes sense she would only think there has to be a practical reason why people wouldn't ostracize him.
Let's be fair, they all seemed genuinely taken aback by Kaya's conclusion and she quickly apologized for it. It's right now very possible for them to have completely missed this disconnect between them without Kaya's input. Yuki's never on panel asked Toru about how he perceives them, so he hasn't lied to them about it in any way she would detect. Similarly, if he's so internalized he cannot have friends and won't allow himself any- what distress is there in him for Kuga to see if he's accepted and embraced it years before ever meeting them? He's also dodged having to attach any label to Sasamori right now, he hasn't even friendzone'd her. Hijiri has called himself Toru's only friend multiple times, but Toru's never explicitly called him a friend to the best of my memory.