The emperor is probably jealous and insecure that she will take away the only friend he has and on the other hand, he probably believes that she is acting fake towards Tori since it seems since the very beginning that he believes that nobles are "evil" and disgusting towards commoners. His anger is directed at her the very thing that is considered everything he hates.