FYI peeps, people have had their children abused by someone they trust in real life many more times than any sane person would want. See church, scouts, school, day care, you name it... any time... anywhere. This pact of silence is a common hallmark of abusers.
So... before going off on the duke now, it's all about what he does once he finds out. Yes, it's upsetting as a plot device but it isn't proof that he's been a bad or neglectful father or bad person.