Hm. I feel like a lot of people are forgetting that if the empress' son doesn't win like she said she'll be trapped there (imprisoned basically). They have said it before and also when they showed that grey haired woman that Noah (the blond) met. She was forced into that isolated palace and didn't really get to come out or much food or anything. In her case they literally forgot about her. And she was only someone that married the last emperor towards the end of his life and she never actually got with him, because she was too young when she was married in I think.
Anyway, the point is they don't let them go because they don't want them to start anything, especially trying to bring back a kid and claiming it's the emperor's or something. But also starting an uprising and they want to control and watch them They have mentioned before how cruel it is and I feel like Cesar said he would never want that to happen to Canaria.