This is somewhat getting tiring cuz iirc these people never really had any time where they genuinely could take a break. Everytime the situation cools down, somehow another wave of enemy shows up and ruin their rest time.
Also I kinda get that he's being cautious about his ability but like since they got more members, that means more firepower but also more people to protect. And since he only trust only a select few of them, that means the gap between their power will inevitably increase leading to the ones actually able to do the job be stretch thin, which betrays the very purpose of working in group. I mean look at them getting their friends whittled down one by one, frequently got kidnapped, etc.
Tbh I don't really remember if there's a limit on how many people he could link to his ability. Suppose there's none, then this mindset of "few elites" will backfire in the long run. Looking at the prev development, those girls legit don't have any incentive to betray him either anyway. And if he won't, he'll have to work extra on leveling them up which, again, back to the first point: they rarely have spare time to level up and train.