So he decided to take care of the late Empress' son because she asked him but neglected his own son while doing so?. And when choosing between them the Duke probably thought it would be better to choose the Prince's side so he'd continue to trust him making it easier to continue looking after him. Of course his own son would rebel if both parties stop showing him attention at such a young age, and also treating another kid better than their own child doesn't help, but it's not like he can leave or run away. But the prince can, and if he did then the Duke wouldn't be able to uphold his promise to look after him. But that's still not a good reason to neglect and hurt your own child, at least the mother is trying to connect with her only son. I wonder if she knows about her husband beating her son until he bleeds, she'd probably fall I'll if she didn't know and suddenly found out.
Thanks for the update ❤️