That's some hamfisted bullshit. "Oh no, not the mysterious gaze at mah back! I can't endanger my two party members!"
Bro, who or whatever that gaze is has already realized that these two girls are close to you and if that gaze has malicious intentions it can way easier do you harm, by using the girls as hostages or whatever. Might as well stay with them. 3 people can better defend themselves than 2.