Honestly. I can understand the distrust for the lords, especially if they actually ignored the problem, while acting high and mighty. Yet we learn after one of these actually tried to fix it, but failed. Now giving up could certainly be considered a betrayal for the villagers, yet I consider unfair especially if the mage didn't receive support from the central government. Mixing people who tried with people who ignored the issue is assholeish.
Also I can understand acting distrustfully, and even mild confrontational, but "Go Back", "Leave us alone" isn't something you should tell a feudal lord. Especially a royal. Even if that country have an extensive anti monarchic sentiment, they know better that to making it manifest. Giving them a 2 weeks limit is even more stupid.
Also what did the villagers have to loose? It's not like they will be more starving then before even if stuff doesn't work. I could understand the villagers menacing the lord to leave if they started collecting / increasing taxes before the problem is solved, but not before.