It does work on neutral parties so long as they've never heard of him before. That enemy assassin was one such case; she'd never heard Streamer's voice up until that point and was completely indifferent to him.
And keep in mind, what Mira used back then was most likely recorded using Kanata's "new" ASMR mic, which I will remind you, was built with fucking mithril of all things! Not 100% certain of how mithril works in this isekai, but if it's anything like what other works make it out to be then that means that the magic he uses gets amplified tenfold. I'm guessing that's how it works, given how, on max settings, he made the freaking goddess climax so hard she went back to heaven in an instant.