Oh wow....his dad must have been hella strong to always win against his son X'DDD Reading the first chapter, I thought this was about an ordinary man & his family XDD
I think the MC's father might've wanted him to live a normal life instead of being distracted by fame/ambition and having his family suffer from it. That's why he kept telling his son that he was just a wanderer after the MC kept losing to him.