Again, just because the emperor and imperial family saved them this time doesn't absolve them of their neglect towards the twins and they absolutely should never be forgiven.
Even if he couldn't get close cause of crazy mom he could have insured that his kids were taken care by ordering the palace maids to properly do their job, and knights to put a leash on their abusive mother. Those don't require him being there.
He could've ensured they at least got basic care and weren't abused. He did not, now he wants to be a happy family but honestly needs to just fuck off with the rest of the princes.