The twins eavesdropped on that conversation, and knowing they don't have ANY ALLIES in the Imperial Palace (the head maid included). They were bullied since young with a drunk mother, neglected, physically and mentally harmed. Had the two of them not experienced another life before and had been mentally mature, they would've been dolls at this point. They have serious trust issues that are well founded, seeing as people out of the blue that had once abandoned them and openly admitted to forget about them through the years suddenly come back in their arms and try to treat everything as if nothing bad actually happened. You're not gonnna be talking about that with ANYONE at all. I'm surprised they didn't even go bipolar at this point.
The Emperor is a poor father figure. He's trying to ease into the role of being a father but is indeed failing for the majority of it. So, he tries to make amends with them by being overly protective and supportive, without actually taking into consideration what the kids actually want. You can see with all their interactions, it is mostly one-sided because he has not only years of experience governing and controlling people, but had also raised three kids who came out well, not considering the needs of the twins to be very special given what he had done. And now the attempt to come close became swiftly undone after the talk with the duke's kid. His ego can't take it, and thinking he wouldn't want to make it worse for the kids, goes get the story from the other side, with NO POSSIBLE WAY of verifying the veracity of what the kid said, since it was just him and the twins, and he with his pride and ego, can't simply talk to the twins.
The story seems worse and worse because you're looking at it like an armchair general. As a reader, you can SEE what's happening on all sides, so you can tell what's going on, what failures of logic the people take.
The characters are well made, psychologically-wise. They all have their reasons for doing the things they do.
If you think they are making misunderstandings because "no sane normal person" would think like that, then you need to change your viewpoint. The twins had serious trauma, serious trust issues. They are NOT normal people. The emperor is also NOT a normal person either, with his seat of power, confidence, and ego.
