The very reason he is calling him selfish is why calling him selfish makes little sense. His conclusion/guess was he dropped his job with the princess known for being selfish to live a better life. He basically concluded by himself that he left because his work sucked. The details might not be mentioned but it isn't a matter of the MC not giving the required context. I guess in a medieval context it is not out of place for someone to think it is a servants duty to serve a noble and especially a royal and they should be loyal even if they aren't happy there. But it is not like we as readers have to give a fuck about whether it has cultural reasons.