If they abandon the adventurers' town, won't the enemy occupie it as their own base? The enemies would even get the ballistae there to defend it. Depending on the enemy's objectives, they might be happy with that alone. If it's a scouting unit, it would serve their purposes perfectly while waiting for the main army to arrive. If they are planning to do nothing but scout, they will just loot the adventurers' town and be done with it. Van is now paying the price for making the adventurer's town so exceedingly separate, located a couple of kilometers away from the main walls.