What I’m guessing happened in the first timeline is FL didn’t trust the Duke AT ALL and actually believes all the rumors and probably treated him very differently than how she is now. That in turn, made the Duke ignore her and pay her no mind since how all she probably did was fear him anyways. That’s probably why he told her she should starting trusting him before he sent her back in the time.