Severe food shortage is associated with increased mortality and reduced reproductive success in contemporary and historical human populations. Studies of wild animal populations have shown that subtle variation in environmental conditions can influence patterns of mortality, fecundity and natural selection, but the fitness implications of such subtle variation on human populations are unclear. Here, we use longitudinal data on local grain production, births, marriages and mortality so as to assess the impact of crop yield variation on individual age-specific mortality and fecundity in two pre-industrial Finnish populations. Although crop yields and fitness traits showed profound year-to-year variation across the 70-year study period, associations between crop yields and mortality or fecundity were generally weak. However, post-reproductive individuals of both sexes, and individuals of lower socio-economic status experienced higher mortality when crop yields were low. This is the first longitudinal, individual-based study of the associations between environmental variation and fitness traits in pre-industrial humans, which emphasizes the importance of a portfolio of mechanisms for coping with low food availability in such populations. The results are consistent with evolutionary ecological predictions that natural selection for resilience to food shortage is likely to weaken with age and be most severe on those with the fewest resources.