@abyssalmonkey I suggest you reread the first few chapters... The only reason the little sister and the brother are treating him better is because they saw his abilities when he used them in front of them. Before that, they were treating him worse than his wife is treating him, going so far as to attack him.
Unlike them, the wife never saw his abilities, but is still treating him decently, defending his status against the opinions of every single elder of the clan.
I'm not saying he should act as a lover to her, all I'm saying is she deserves way more respect than he's giving her.