If you look at the Emperor's reaction face after she says "Have you fallen for her", that seems to be the line that really brought it home to him and killed it. Up until then he seems merely curious and wants to protect his new toy, but his actions are already noticeable enough for his sister to come complain to him in person. He's just realised that if he goes any further than this, he's gonna end up being known as the foolish emperor wrapped around the finger of a mere lowly maid; actually not even a simple maid, but a thot that already "played with" with 3 of his sister's "toys". It's one thing when he thought he was just playing with a new toy, but she's just made him realise it's a whole different game if he's gonna declare that he's in love with this kind of maid.