Hmmmm intersting, Endeavour was not a "total" asshole to his own son after all, but it took him too long to notice and rectify the mistake. If only he had shown more fatherly love to Toya even after Shouto was born, things could have turned out differently.