I will never like the emperor. So what if he was fooled and used by the MCs' mother ? He wanted to kill them as babies and then had so many chances to redeem himself but he always ignores them. If he made contact with the kids after she died then why didn't he kill her before ? Or why didn't he negotiate with her in exchange of raising the kids himself ? She was obviously materialistic and didn't give a shit about them so he definitely could've negotiated with her.
The kids are so damaged even though they are a lot older mentally, so imagine how bad everything must have been. Also, every misunderstanding is his fault. So he heard the kids want to return and then didn't take their opinion at all and was like "yeah let's get them back to the place that traumatized them without considering their opinion because some asshole told me so and I'm an imbecile fucker whose mind has ceased to function a long time ago"