That area of the world building is questionable, yes.
That said: they're called borders, but what's on the other side is monster land, not another country. So being a lord there is about dealing with monsters, not foreign relations.
The Stewarts are a family of meatheads. Even the scholar...