Mage was a bit cold blooded, but they were right. Before Aria, they could only do floor 1 and 2. Now, going up from floor 5 while shielding and moving a paralyzed member? Too hard and dangerous.
Plus, swordsman NEEDS a sword to be able to fight his way back, he couldn't just left it, even more...