I wonder if Colette had idolised Harold in her mind, over the years. Her only personal experience with him was the brief moment when he saved her and Clara's lives. It's possible his rude behavior hadn't left much of a lasting impression, compared to things like receiving the necklace. Clara would know more, having served in the mansion, but at the end of the day, Harold saved her life and went out of his way to arrange a new life for them, so I doubt she spent time telling horror stories to her daughter. So, if she had indeed formed a very positive image of Harold, it's no wonder if his attitude now made her cry.