Yeah, don't go looking for logic in what was originally a web novel, more than likely written by some rando Japanese teen. I mean, I get it that the whole thing is basically just an excuse for some light hentai, but even so it's still worse than I'd expect.
It makes sense if you go by the rules of their world, which is basically that some people have what are essentially superpowers, and others don't. How many farmers with pitchforks would you need to replace the battle power of one mage? A better manga could probably deal with this with some nuance, but this is a crappy generic isekai, so of course it chooses to just ignore its own worldbuilding when it suits it.