It's so nice to see that the Shadow Worlds have different ways to them. Some require pure strength, some require wits.
I bet that defeating barbarians would make it a lot easier for the Lord to solve the homeless ones problem, but by getting the extra quest completed Desir now has a way to get much more information. It opens new ways for the main quest to get complete. Maybe you just need to get something/someone from both homeless ones and barbarians to end it all peacefully. Maybe someone somewhere has an artefact you could gain besides the mana gloves. Maybe by completing all extra objectives in that world, the next one will be easier to complete? It's really great that it didn't just turn into "beat big bad evil guys".
I wonder which shadow world Desir entered for the first time? From the few first chapters I got a feeling that he only got involved truly like 3-4 worlds later, when more than a half students/population died and they had no other way than to use even the weakest ones.