Please, please don't let it be much more complicated than this.
We don't need another Habsburg in development, even if it's just manga; let's keep it in eromanga territory instead.
Ok so new theory gang, dad was an harem protagonist and was/is going through harem members like crazy but turns out he's shit at maintaining relationships or the women all agreed to spend X years with him and then divorce him.