Duke sees Stella only as his little sister but she clearly wants his dick, so obviously she would make all the staff to hate the Duchess. And the Duke is just overprotective and trusts everything Stella says. And since she's handicapped, it might be too hard for her to travel, which explains why...