Yup, and I can prove it: Ancel Keys had a tiny dataset — but that didn’t stop him from leaping to big conclusions. Nina Teicholz wrote about Keys’ problematic data in the terrific book The Big Fat Surprise, and I just came across an old paper that backs her up.
The paper appeared in a 1989 edition of The American Journal of Clinical Nutrition and was (of course) based on Keys’ famous Seven Countries study. You’ll recall that Keys supposedly recorded what people in seven countries ate and then followed their health outcomes for several years.
Here’s a description of the study’s design from the paper:
During the base-line survey 13,000 men, aged 40- 59 y, were medically examined. Information on diet was collected in random samples from each cohort by use of the record method. Detailed data on food consumption patterns have been published only for 9 of the 16 cohorts. Therefore, the food intake data were coded once again into a standardized form by one person. Then the foods were summarized in a limited number of food groups. The average daily consumption per person of these food groups was calculated for each cohort.
So Keys had food records, although that coding and summarizing part sounds a little fishy. Then he followed the health of 13,000 men so he could find associations between diet and heart disease. So we can assume he had dietary records for all 13,000 of them, right?
Uh … no. That wouldn’t be the case.
The poster-boys for his hypothesis about dietary fat and heart disease were the men from the Greek island of Crete. They supposedly ate the diet Keys recommended: low-fat, olive oil instead of saturated animal fats and all that, you see. Keys tracked more than 300 middle-aged men from Crete as part of his study population, and lo and behold, few of them suffered heart attacks. Hypothesis supported, case closed.
So guess how many of those 300-plus men were actually surveyed about their eating habits? Go on, guess. I’ll wait …
…
And the answer is: 31.
Yup, 31. And that’s about the size of the dataset from each of the seven countries: somewhere between 25 and 50 men. It’s right there in the paper’s data tables. That’s a ridiculously small number of men to survey if the goal is to accurately compare diets and heart disease in seven countries.
But wait … so far we’re assuming the dietary records were accurate. As Teicholz pointed out, Keys took one of his food-recall surveys in Greece during Lent, when religious Greeks abstain from animal foods. I’d call that a bit of a confounding variable. And then there’s this, directly from the paper:
In Crete the villages involved were Agies, Paraskies, Thrapsano, and Kastelli. In Corfu the villages were Ano Korakiana, Skriperon, and San Marco. About 30 men were involved in each dietary survey. However, the original 7-day records were no longer available.
No original records?! So you dumped the study, right?
It was therefore decided to reconstruct the diets of these cohorts on the basis of results of the dietary surveys mentioned in a publication by Keys et al.
Uh … so you swapped in the results from an earlier paper. Okay, got it. But tell me we’re at least talking about a genuine dietary survey here.
When no information about the consumption of certain foods, eg, fruits and vegetables, was available food balance sheet data from Greece in 1961-65 were used as a substitute.
Head. Bang. On. Desk.
Getting the picture? Keys followed the health of more than 300 men from Crete. But he only surveyed 31 of them, with one of those surveys taken during the meat-abstinence month of Lent. Oh, and the original seven-day food-recall records weren’t available later, so he swapped in data from an earlier paper. Then to determine fruit and vegetable intake, he used data sheets about food availability in Greece during a four-year period.
And from this mess, he concluded that high-fat diets cause heart attacks and low-fat diets prevent them.
Keep in mind, this is one of the most-cited studies in all of medical science. It’s one of the pillars of the Diet-Heart hypothesis. It helped to convince the USDA, the AHA, doctors, nutritionists, media health writers, your parents, etc., that saturated fat clogs our arteries and kills us, so we all need to be on low-fat diets – even kids.
Yup, Ancel Keys had a tiny one … but he sure managed to screw a lot of people with it.
