I swear I've seen this concept in another sci-fi work.
But yeah the idea here is mostly that you lose out on the time spent with others (although this galaxy may be so advanced that time too is mostly irrelevant so it's not likely they'll die of old age)
So it may just, 100%, be you missing out...