You are mistaking some points regarding Yuto.
The reason his Charm power was ignored is because it's not completely developed. His powers were less about drugging girls, and more about creating a favorable opinion in the minds of others.
What the person affected by it did with this favourable opinion was upto them.
Aka., unless those people were themselves attracted to Yuto, his Charm wouldn't have affected them. This is what happened with MC's Step-Sister, Childhood Tomboy friend & his Older Neighbourhood Girl. All of them had problematic personalities and were attracted to Yuto more easily.
This is why it is also mentioned that a lot of people didn't really like him either including his own parents & brother. I mean, all he needed to create a favorable opinion is touch them, and it really was restricted by gender as he could create favourable opinion is guys as well.
Also, Yuto's Charm powers were broken away as soon as he came back from Isekai, as the power he brought back caused a ripple affect and disabled every weak skill/effect around him when he appeared. And since his return Yuto haven't been able to Charm any new girl as he was himself confused about the situation.
As for why the Adults who knew about Spiritual powers didn't act against Yuto for so long, well its basically because they were given false info.
Basically the Spiritual Agency sent girls & women undercover to investigate him, and they reported that his ability wasn't that big to require focus. But in truth, all of those investigators had ended up developing favourable opinion about him without being charmed and even slept with him, which is why they lied in their report (well, ignoring Yuto's Charm ability, he is actually a very good looking guy enough to be classified as a model, even though he apparently has a small dick).
And to add, until before 2023, minimum age for consent was 13 in Japan, meaning they weren't considering as undeage until recently atleast as far as sexual act was considered.
Plus, it's actually quite common for girls & boys in Japan to have sex when they were around 13 (based on a recent video on YouTube were a Japanese person interviewed japanese women on when they had their first time).