@Knighto : Well, according to the page where Yuria explained it:
https://mangadex.org/chapter/10201/4
The more people recognize you as their leader.
The key is that 'recognition'.
Even if many people are still within Rigar's territory, how many of them after this refugee incident will still recognize him as their leader?
Or how many will come to believe that Almus should become the king instead?
These people doesn't even have to be in Dibel or Ash territory, the more neutral nobles will definitely start to think about what to come.
And the refugees each are worth 2 'points', one lost for Rigar, one gain for Almus.