*The human body just isn't that weak.*
It mostly depends on the mutational load that a person has. Normally, before the Industrial Revolution. Only less than 50% of the population can only pass on their genes, as such, accumulation of the mutational load is very low. It is only when the industrial revolution, that the accumulation of the mutational load is possible to the point, that it became very very high and maladaptive traits started to appear at a much higher frequency (i.e: infertility, left-handedness, genetic disorders, etc. ). Majority of the mutational genes are recessive and inbreeding actually make those recessive mutated genes to appear at higher frequency but... It is also a way to actually decrease the mutational load.