It was probably the father but he has some memories because she was dead by the time the spell was done. So there was a spillover of memories that he got some
What I love is that idiot didn't care about trash talking to a foreigner all day and spend time with him. But the empress have a business talk and he gets all angry.