As of chapter 450 in the WN (earlier, but that's where the English translation is at right now), Haruka questions this, and very slightly suspects it may be the case.
In the WN, Haruka actually launched a fireball at him as he escaped and just wasn't able to land a square hit. He flew away...