Please do recall that the act of violence here in the manga was the act of a very few number of people. Those people were exiled by the rest of Phyto's people. And Phyto herself was given to Lerunda's people as a hostage in both abject apology and a promise that it will never happen again.
It's no good to conflate the story lines of adaptations once they begin to diverge. Compare and contrast, sure. But the manga is obviously taking a different route than the LN, so you can't judge the characters in the manga based on what happened in the LN.
The refugees are often portrayed without eyes and distinctive features, which essentially portrays them all as a collective. It is pretty difficult to judge the actions of a few when they are all interchangeable.
Also, Roma has been removed from the story in the manga to keep track with the source material. For all intents and purposes, Roma's current role is no different from being dead.
But even then, the refugees had taken a hostage to demand being welcomed in the village, injured Roma gravely despite his attempts at helping them at the cost of the village's safety, demanded to have Lerunda, the small child that saved them, present for negotiations over a hostage, and even sent another child to be a hostage!
Even if we separate the manga from the source material, those refugees still represent danger to the village, Lerunda and themselves nearly every time they're shown!