@riflow From what I could tell through his conversation with the servant that came to him, pleading that he convince his sister Laelin to reconsider firing him, I think it was because he wanted Lilian herself to dismiss them. She wanted Lilian to be properly recognized by the servants through her own efforts and didn't want to coddle her, but since Laelin ended up doing it that plan is now out the window. I think he figured that nothing truly bad would happen to Lilian within the household, especially if he is keeping an eye on things, but if Lilian is in the palace then he can't be there to protect her should things go too far and thus he's panicking. As a result, his affection for her is going haywire and it seems as if his personality did a partial flip from uncaring(since he knew about the harassment and did nothing that we know of) to overprotective(such as trying to get her to return home and freaking out at the idea of her marrying the "Tyrant" Emperor).
Simply put, the brother was in control of things back home and therefore could protect Lilian from the worst of the harassment from the servants as well as maybe their father's abuse(we don't know how physical the father typically got with the elder siblings). At the palace, however, he has no control and can do nothing to protect her. In other words: Overprotective older brother that wants his adorable little sister to be safe but also strong on her own two feet is freaking out because he can't keep her safe in the little bubble he made for her at home.