In these types of stories, it's best to take things at face value.
If the loli states they are being abandoned, and the 'big' bro tries to tell her there are wicked people in the world. It's more than likely that he abandoned them.
Sure, he may come back and say he was leading them away, to "save face". But the villains in Manhua are always over-the-top, likely leaving them to die because they witnessed his failing.