my pet theory is that the farther and the time mage used the time-reversal magic with each other but the farther chose not to restore his own memories but the MC memories instead. Personally, I think it was done out of cowardice as he does not want to remember that he killed his own daughter but...