So why is it that the castle/walls knew about the duchess being fake, but didn't see the duke dealing with any of the info of her being fake? You can't have them be the eyes and ears of the building ONLY when it's convenient.
Either they know what's going on, or they don't.