It's really not his fault in the first place bc the empress made him marry Yan An & his mother is quite ambitious... The bad thing was that he was keeping secrets from her & continued to meet with the 2nd princess. Also the 2nd princess should've known better, it's not Yan An's fault too.