My guess is, 4 years later something happens that needs all adventurers to gather, even retired he would still be A ranked which would force him to come. Something like that I guess.
They left and came back 4 yrs later? Don't get why there was a time skip in the first place most of the time skip happen a few chapters later on so why this early?