While holding back mere villagers should be easy, isn't the logical conclusion that MC merely takes the confusion ""debuff"" himself, stopped by all his nullifications? The fact he took his ex-parties problems without their knowledge means he doesn't need consent.
