If you did read the first chapter, you will know that he spent 30 years of his life till his passing on trying to use magic even as far as staying virgin.
Take away his only salvation or belief, he will be nothing more than he (claim to) hated the most, "being a normal human being".