Mostly because he views Tomoe and Mio as family, like they're siblings or cousins. Unless you're a specific kind of degenerate, that isn't exactly appealing. There's also a certain amount of power dynamic: There's a level of coercion there due to the master-servant contract that he isn't...