This might be a "guess I didn't trust her as much as I thought" type of plot thread. People have trouble understanding their own hearts. None of his inner monologues ever talked about doubting her. He always accepted her past but, he might not have realized he was burying his trust issues. That...