@Groceries
hard to say. Seems like a lot of the problems come from the weird pacing of the manga adaption rather than were the light novel goes. If you ask me, what's coming up was a really good part of the novel and a major improvement for the plots development, world, and the characters. A character that's the leader of glasses group that's about to be introduced, along with the rest of them, did a lot to balance out the cast and the stuff that's happening now and later on is much more important for the plot than what was going on with the king and church in the first arc. It also does a lot to ground Naofumi in the world, opening him to the idea of building a place for himself there.
However, all that might not mean anything if the adaption keeps being wonky. I mean, in the light novel, you could tell that Nao was opening up to Ost and learned some significant things from her. Yet, I don't really get that impression from the manga. The fights also don't feel like much is happening here while they felt pretty significant in the light novel.
Basically, at this point, I would say that anyone thinking of dropping after this part probably should drop the manga but try reading volume 7 of the light novel. It's the section of story we just covered so if you like that version better, you should just keep reading the light novel and ignore the manga.