Yeah, the duke was scummy in the past. But I don't care. I've read enough stories about scummy dads from "Who Made me a Princess" to trashy MLs like "The Abandoned Empress". Overall, what matters to me is if the Duke can actually work to become a better person. Even if he was scummy before, being played by Cosette like that and being a shit father, what matters now is whether he can change fate by becoming a better person.
Even if his future self reversed time just to undo a fuck up rather than out of concern for the daughter, what matters to me is the present duke's opportunity.