Having that kind of body at high school just unbelievable. Man, he could make a living as MMA fighter or a boxer. I wish the author gives more flashback after Ryouma's mother died though. It would explain the other skills beside martial arts, physical and mental resistance.
Yes, I do think he is a bad father but you guys blame the father too much. He is a kind of man that only care about his work. Do you think that kind of man would want a wife or kid? Nope, they would be a hindrance to his work. There's a thing called pre-arranged marriage here in Asia. His ancestor probably pushed a wife and a kid into his life to keep the dojo and family tradition running. Well, at least he managed to fulfill the minimal obligation of a father, even though that's barely enough.