"Young'un", if you let the other go without a fight, that means only one thing: You did not love her/him enough to not give up a possible twist in your own future. In other words, you were a coward, choosing the easier route. (Your argument was based on exactly this, giving up the whole relationship before it even started) While usually cowardice is not a bad thing in itself, if you decline most possibilities (and what stops you from doing so with similar excuses like the original quote?) from yourself, in the end on your death bed you will not have anything you can remember. And I think that is a much worse possibility than the one you could have originally / the one you described.
On the other hand, to continue with your example, what if the elf and the human have kids? Then their love bore fruits and it will remain as a memento of their love forever, in the end probably in the form of a full family lineage starting from just one human long ago the still living elf loved. And why do you think it would be pain? Why could not it be a sense of accomplishment, or joy? "We made this possible, without us there would not be any of this".
IMO if one really takes a quote like the above one seriously, that person have to "jump out of that well that keeps the frog from seeing the bigger world".
And of course this was only a simple / single part of the bigger picture.