@gum_rummy
It's really necessary, for two reasons.
One, Naofumi is paranoid as fuck (see page 18, where he says, "Everything was to mislead me... to ridicule me."). In that moment, he's probably thinking that Raphtalia too wanted to deceive him, and set up the duel in order to be freed...