If you pay attention, they do mention they destroy everything before leaving and the military has already been dispatched.
Running away, even if it's an option, would lead to a vagrant lifestyle again for a long while until they are able to get another house - it would also mean that the swords...