In response to sinkingship, my impression is that the Harem are women who want to marry him. He can get to know them, then select one to be empress. Except he won't even go visit them. They definitely aren't his wives. The girl that caused so much trouble, then saved the empire, was his first.