This sudden serious turn makes me think the manga is closing on a axe or the author was told to make it more interesting after comedy and fan service didn't make it more popular...
I'm guessing the hero party was seriously damaged/crippled/someone died and mask demon used his power to make them return to before it, that messed up with their memories.
But the dialogue is too cryptic, is not really clear why Toifa memories were messed up by it aside that on a living organism it likely creates brain damage.
I'm guessing that the thing about the priest not having it was that he is immortal for some time magic/curse and can't be affected by it, nor take away nor changed by giving it.
To be is still lost, the mask demon lives by taking time in some ways, and can fix things by expulsing it from his body, the closest thing to it for a normal being is throwing up.