I agree with @Oukishi
He has no right to decide whether the women who marry the dukes are suspicious or not.
Rather than giving up into the doom, it's better to try every possible way to save the duchy, even if they might taken advantage after that.
However Astina is the saviour, the duchy...