I have no idea, why manga never mention it, but in novel it was stated in very first volume, that any of them could return to their ordinary life if they wanted to, but at cost of their powers and memory would be sealed up by goddess.
I just briefly checked ch.39 in raws, and again they...