@saku_mochi you have to consider the setting or period, Caidel's father may not be the ideal father but the man still performed his duty and gave him a home and food on the table. The treatment only got worse when he found out his son was getting acquainted with Ellen (but lets consider the abuse *as weird as this sound* was tolerable at least), heck remember Caidel's mom taught Ellen that while her husband wasn't the perfect man, they needed a man to protect them back then.
Additionally, once the villagers all mysteriously died (sure, we can consider the ones who chained Ellen at least got their karma, but what about the rest? Other kids in Caidel's age. They certainly couldn't do much harm on a lone woman, *other than probably throwing stones at the creepy half black colored eyed woman*, moms who at the very least didn't want their kids to feel frightened on what they considered "different"), remember Caidel was sent to an orphanage where he had experienced something even more terrible (older kids bullied him and the adults scolded him) than his father's abuse and it was only because the Earl saved him (bear in mind, we don't know how long he stayed in the orphanage but we can consider him having thought of being betrayed for YEARS doesn't mysteriously disappear overnight. Much more since again, Ellen accidentally caused his father's death *even if the bastard started it*)
Edit: I might as well recommend everyone to reread the prologue just to see Caidel's POV on the orphanage abuse