Well, another thing to keep in mind is the MC is such an outside-context problem for everyone else that half the reason they're freaked is because of his very existence, not just his supposed power. To the rest of the world he just suddenly has this immense power out of the blue. This regular kid from a (IIRC) regular family, essentially goes from an NPC to Let Me Solo Her in the span of a day. It flies in the face of their every understanding of the world they live in. It would be like a person in the real world just suddenly being able to fly for no reason.
Between that and his desire to keep a low profile meaning he's concealing his power as much as possible, for some it's easier to believe an implausible explanation like getting a lucky shot, using some underhanded trick, the daughters misremembering or screwing up, etc. And the mother is clearly the only one with her head on straight enough that she doesn't immediately dismiss what by their world's rules should be impossible, but that doesn't change the fact that aside from her daughters' recounts and (remind me if I'm misremembering) sensing his power during his fights, he's still a massive unknown. For all she knows there's something to the accusations that he's posturing or that his power isn't all it's cracked up to be.
She also did try and request a spar with him, which he turned down. And, yes, it was wrong of her to force the matter and attack him regardless. But that's how manga/anime logic goes; the only way to test someone's power is to (at least appear like) you're going after them with intent to kill. As for asking for his help and explaining the situation first... yes, that would have been sensible. But she's going against her family's traditions and responsibilities- and plotting against a literal centuries-old god besides- to save her daughters. This is a huge gamble; showing her hand to someone who might not be all he's cracked up to be risks revealing her plans to people she'd rather not have know.
But by forcing the matter she forces him to show what he can do without having him pity her or intentionally underplay his powers to make her believe that he's not strong enough to help her. Which gives her an accurate gauge of his power, since as far as he's concerned he's defending himself against another crazy bitch trying to kill him. Which also gives her an idea as to how his power matches up to the Mountain God by virtue of the ease with which he defeated her.
I can concede on the fact that fighting him to learn how strong he is, is reasonable given your reasons. I still don't think she's a very good litmus test considering she's weaker than the god she wants him to beat, but I can understand the intention here. The issue I'm having is that if she really wanted to fight him, then there are more risk-averse ways to do that, and another problem is that her actions before the fight don't make sense because of it.
Because you're absolutely right, a random person pops up with power that she immediately believes could rival a multigenerational family that specializes in magic? Ain't no way she's just going to let that slide, but she's also smart enough to know that this person is still extremely powerful and thus extremely dangerous—she is pretty confident in that, we see her talk about it often. So she stays cautious and does damage control where needed. This works towards her initial goal of making sure this sudden, extremely powerful, unknown threat doesn't fuck shit up (lol XDXD), and at some point she
does realize that this person could potentially be strong enough to take out their patron god, which prompts her to check to see how strong he really is... ergo, fight club admission.
But then you think about how the attack works towards that previous idea. Sure, it might give her a better understanding of whether he can kill the god. But he's still a 'sudden, extremely powerful, unknown threat'. All that caution in keeping an eye on him and apologizing when they attack him twice, is completely thrown out the window by directly antagonizing him... again.
So now she knows he has the power to help, but she also make him less likely to do so. Not a great trade off. The risk of explaining first is he
could walk away knowing they're planning to kill a god, but they don't need to explain
everything, just the minimum needed to convince him—they could even say the god is a demon and, though it's a lie, if he says no and tells anyone then he's just going to get a, "Yeah that family kills demons, so what?" XDXD. If he finds out the truth he could get pissed off, but more pissed off then having your life threatened three times? Not likely.
The only reason someone would make such an unnecessarily risky decision like she did here, is if his help was absolutely guaranteed. Which it is, but only because it's fiction, not because it makes sense. My version also isn't very cinematic, doesn't have as many fight scenes, and also wouldn't necessarily translate well to a story like this. Might be able to but I'm not a writer, just an ordinary citizen with far too much to say lol. XDXD