Despite the growing harem, he's actually quite prudent. There have been and are a whole lot of beautiful women around him that he has no interest in. Certainly, he's not satisfied with only a single woman, but he avoids dalliances almost entirely. I can only remember two cases where he had no interest in a long relationship: Bi Luo and a nameless maid in a mansion he was visiting and the host wanted to entertain him. I think it might have been when he was living in Xue Yue's brother's mansion, though I could be remembering the place wrong. There was also the incident with the charm demon, but that's too muddy and under forced circumstances to include here.