He could have said "You were my lover back then, but that time Riize was dead. So ...."
That's not a good answer I guess. Since it would unintentionally make the elf seems like a 2nd choice and it won't change until Riize died or changed. It would create unnecessary fights or plot