I feel like it says more to the "side" story that you think there's 0 story to them once we're done with Shiro's side in the manga.
I watched the anime while the translation for the manga languished, and I much prefer the flow of what we're getting in the manga instead of the switching to people I don't care about in their standard RPG isekai situation.
Yeah well, the problem with this series is that the story and the worldbuilding come from the side characters. White is mainly here for the comedy. Ironically, it's when White starts doing stuff that makes the story moves forward that she starts to lose the PoV as well.
That's also the reason why the Manga feels like the story goes nowhere : all relevancy to the story was cut alongside the side character's PoV. And I'm not even talking about Shun, I'm talking about Ariel, Merazophis, Wrath, Dustin, Potimas, Julius, the old magician (if you think the threat of the other Kumoko came out of nowhere, that's normal : the old magician's PoV was completely cut), Agner, Bloe, Balto...
No wonder you don't like the supposed to be good guys, when the Manga made them so dirty (and it doesn't help the author doesn't know how to draw humans).
Also, the situation Shun is in, is not typical in an Isekai.