Most nutrition researchers are forced to collect their data using a notoriously unreliable scientific instrument: the human brain. It’s hard and expensive to conduct rigorous nutritional experiments where you know through direct observation and measurement exactly what people are eating. Instead, most studies are conducted by asking people what they ate.

How far off are the data? Researcher Edward Archer and a pair of collaborators published a paper in 2013 that attempted to put a number on it. They looked at almost four decades of results of the National Health and Nutrition Examination Survey (NHANES) — the big U.S. government database on diet — and compared what people said they ate with how much they’d need to eat simply to stay alive. Their findings: "Across the 39-year history of the NHANES, IE data [that is, energy intake data — how many calories were consumed] on the majority of respondents (67.3% of women and 58.7% of men) were not physiologically plausible." That is, if people ate what they said they ate, they would have starved. In other words, more than half the data were demonstrably wrong. And that’s not even counting people who were over-reporting.

People miscalculate, misremember, and straight up lie on dietary surveys. How can we trust nutrition research when we don't even trust ourselves?

Now, there’s wrong and there’s wrong. If you have an otherwise functional clock that just happens to be set ten minutes slow, it’s no problem to adjust the incorrect data it provides you at any given moment. The question is whether faulty food data can be similarly manipulated to make it useful.

And what happens if survey-based data suddenly become unacceptable? A lot of nutrition researchers will quickly discover it’s a lot harder to find funding, conduct studies, and publish the kinds of articles that provide tenure, job security, and prestige. And these endangered scholars are the peers who pass judgment on the articles that appear in peer-reviewed journals. They have a powerful incentive not to rock the boat.

Read the full article about why nutrition surveys being untrustworthy by Patrick Clinton at The New Food Economy.