@sheepdoge A number of things. He's not very interesting as his whole life seems to circle around Mary. There's little else to him. There's also too much history between them, and all of that history he has been her servant. While he barely acts like a servant and might occasionally seem like a friend in the eyes of a stranger, it, nonetheless, means the inequality has shaped their relationship from the beginning. Furthermore, the fact he doesn't act like a servant, much of the time, yet can't be like a friend either, has twisted his personality to the point where I can't really judge his character at all. Could you actually imagine how he would act if they became like a couple, not a master+servant anymore? I haven't got the bloodiest clue what he would be doing and how he would be acting differently (if we ignored any romantic moments).
While the above, I hope, could be considered half-objective, subjectively I just don't like him that much. He's kind of condescending to Mary. Maybe Mary deserves it by being weird, but he's still her attendant for the time being, so he should have as much respect for her as for the duke, or at least he should behave like he has, even if he doesn't genuinely do. The biggest reason is, though, what I began this whole rant with: He's not an interesting character in my eyes. The story never introduced any alternatives, which is a pity.