It's not that the guy is agreeing with the bratty siblings, but he has a lower standing than them and has to be careful with how he behaves. Since they addressed him, he has to try and find a way to respond without coming off as impudent. The fake smile is very expected of a noble.
Even if he really did look down on her, he completely changed his mind at the end when she gave him something to think about.