They didn't move away not because they didn't want to, but because the colour poison (which had contaminated the drinking water of the Gardner family since the 2nd chapter) had made the whole family too numb to care anymore. Notice how Nahum refused to dig a new well like Ammi had suggested, and also refused to call for a doctor even after 2 members of his family went mad, and shown signs of being possessed by the colour.