@DeadintheWater: It's a story about a bunch of OP characters having wacky fun time adventures while some sensible people are nearby going /facepalm and try to limit the damage.
Even if somehow he has something to overcome the amulet, chances are it'll just backfire.
Like, from how the previous girl was acting, his charm is clearly not the absolute obedience type but more of the type that make people trust what he says/like him more seeing as the group could convince her to help them out.
So let's say he use that charm on Rein, maybe she'll actually get ANGRY at how dare he tries to make her doubt her devotion to Hans.
Then Mitsuba probably run by and casually beat the shit out of everyone, while being immune to the charm simply because she's TOO STUPID.