I have read too much isekai manga because I immediately thought the man had got isekaied, but then found an opportunity to return much later, after becoming a king in the fantasy world. A temporal disparity between the worlds could have explained why he seemed decades older than his former wife and son.