Nobody in this manhwa has it easy. Clarisse's case goes without saying. Duchess is a mere servant, maybe not even a proper lady-in-waiting, forced to act like the duchess for the sake of her mother. But moreover, if her real identity was revealed, it would be a huge scandal and her own head would be rolling as the first thing. The duchess's evil maid is rotting in prison, most likely only waiting for the day she falls ill and dies, so she definitely wants to do something to get out. However, the question is how? If she reveals everything, she will also reveal herself as a part of the conspiracy and lose her head.
The count outrageously offended a duke by sending a servant instead of his own daughter, so I don't think he's sleeping that well, either, especially since he hasn't got any news of a child being born in the ducal family. If there was a child, chances would increase that even if the duke learned the truth, he would keep it hidden.
The duke himself has believed for three years that his wife is living in utter sadness due to a forced, unhappy marriage. So, he hasn't touched her. He's not living that happily either, consequently, believing his wife is in love with another man.