I think we just don't see the moments when he favors the other prince. For good reasons - nobody wants seeing that.
In earlier chapters it was suggested to him that the best revenge on the Empress and her kid would be to take away what they want the most, which we all know is the crown. And he...