Yeah, he's definitely doing nasty shit, the fundamental issue is that they have to take him (and the good crops) away without really being able to explain to the villagers why; or even if they did, there's a risk of backlash anyway because some people have the "fuck you, got mine" mentality.