@wildyukari91 He did reject him, but by SJ being a public figure, it'd be easy to catch his departure. He can't stop him from coming of his own will either.
In a sense, it's like; "I'm not following you, I just happen to be have the same destination."