Back in the chapter when the fireball was put up in the air, I didn't really get why the mercenaries would desert, but in this chapter I think it was made clearer.
The spell was strong enough even while charging that it just overloaded every magical defense - this means it would probably also be felt in the bones (or souls) of the would-be targets.
Add to that the fact that they don't know it was interrupted because nobody wants to use it - it could have been just kind of a warning shot (like when someone pumps their shotgun) - or it could have been a failed casting.
So to remain in the army would mean betting your life that 'it was a failed casting or illusion - and they will not be able to do a real cast' so the pessimists are abandoning the cause. (beginning with those who can simply move to another country and not face retributions from their employer).