More likely it's improvements such as magic that have affected mortality rates.
In 1800, a bit over 40% of newborns didn't make it to their 5th birthday. This is averaged (it was closer to 25% in Sweden and over 50% in places like India) and doesn't include other 'ends' to a couple's fertility like one of the two dying early (like, say, in childbirth).
So, say you had a couple, that between the wife's ages of 15 and 40 had 12 kids. First we kill off five in infancy so we're down to 7. Then we add in diseases, accidents, possibly attacks as well...
Now say you live 500 years and the females are fertile from 65 to 300. Even if they have kids 1/3 as often (so a kid every 6 years instead of one every 2 years). you're still looking at a probable base of around 39 kids over more than two centuries. But even WITH things like curative magics, that's a LOT of time before your kids can make you grandparents, for things like accidents, disease, war and other factors to thin them out. That excessive time needed to reach adulthood could mean a rather harsh reality to the Elven peasantry.