*sigh* villains in these kinds of stories always forget the first rule of racketeering: you can't take money that isn't there. If you squeeze your prey so much that it can't recover, then you lose any future income from them.
Especially in this case, the town's population, while large, is not really renewable (children take too long to reach financial independence, with policies actively inhibiting that. And this hardly seems like a prime target for immigration). Lordy here will run out of peasants to extort doing this.
Short sighted... Well, I guess people are like that.