Critique of Dietary Surveys in Nutritional Epidemiology

Executive Summary:

A recent study published in Nature Food highlights a significant issue with nutritional epidemiology research: the unreliability of self-reported dietary data. By comparing survey data to energy expenditure measurements, researchers found widespread underreporting of food consumption, calling into question the validity of numerous studies linking diet to health outcomes. While the problem of self-reported data is recognized, this study quantifies the issue and emphasizes the urgent need for improved methods to assess food intake.

Key Themes and Findings:

1. The Problem: Widespread Underreporting in Dietary Surveys

• Traditional dietary studies rely on self-reported data from questionnaires or food diaries, which are prone to inaccuracies due to memory lapses or a reluctance to report unhealthy food choices.
• A new study used Doubly Labeled Water (DLW) data to develop an equation predicting energy expenditure based on measurable factors (sex, age, weight). This equation was used to assess the validity of self-reported data.
• The study found that in large databases like NHANES (U.S. National Health and Nutrition Examination Survey), more than 50% of adults reported energy intakes below what their calculated energy expenditure suggests. In the UK’s NDNS (National Diet and Nutrition Survey), this figure was over 60%.
• These findings suggest that a large portion of dietary data that is being used for research is flawed.

1. Impact on Nutritional Epidemiology

• The study’s findings raise serious concerns about the validity of thousands of studies that have used self-reported dietary data to establish links between specific diets and health conditions.
• The authors argue that “many studies of nutritional epidemiology that try to link dietary exposures to disease outcomes are founded on really dodgy data.” – John Speakman.
• This could explain why nutritional studies often produce contradictory results, linking a given food to health issues one month, and then absolving it the next.

1. The Doubly Labeled Water (DLW) Technique as a Baseline

• DLW is a rigorous method for measuring energy expenditure by tracking how the body uses oxygen. By comparing the actual energy expenditure with reported food intake, researchers can detect underreporting.
• Previous studies using DLW have indicated that people tend to use more energy than they report consuming, suggesting underreporting. The new study confirms this at a larger scale.
• However, the study’s authors noted that the DLW derived equation may not accurately predict energy requirements for all populations, particularly athletes and pregnant women.

1. Criticisms of the Study and Counterarguments

• Some researchers, like Walter Willett, a nutritional epidemiologist at Harvard, have criticized the study, arguing that DLW is not a perfect measure of energy expenditure and that the problem of misreporting is not severe enough to undermine well-conducted studies and policy recommendations.
• Willett contends that DLW measurements fluctuate and are sensitive to diet and activity.
• The U.S. National Center for Health Statistics (which oversees NHANES) acknowledges the existence of underreporting but defends their data as valuable and important and emphasizes the steps they take to maintain data quality.

1. The Need for New Methodologies

• The article emphasizes the urgent need for new methods to accurately measure food intake.
• Several potential approaches are being explored:
• Photographic food diaries: Participants photograph each meal for researchers to analyze, although this is imprecise and relies on self-reporting compliance.
• Wearable cameras: These would track participant’s consumption but are not scalable.
• Biomarkers in urine: Scientists are working to identify biomarkers that could reveal what someone has eaten.
• The paper’s authors hope their derived equation can be used by researchers to better estimate and account for misreporting.

1. Limitations of Current Data

• Researchers acknowledge that the lack of specificity in self-reported data makes it difficult to know if underreporting consists of people omitting specific foods, such as unhealthy ultra-processed foods or whether they simply underestimate the quantities they consume.
• Lindsay Jaacks noted, “We don’t know if the ‘missing’ food and drink is ultra processed food or fruit or lunch meat or yogurt or sugary milky coffees,” or if people just underestimate how much of each food they consume.

Quotes:

• “If you want to try and set policy around food based on this type of data, then obviously your policy is fundamentally flawed to some extent.” – Gary Frost, nutritionist
• “This [sort of data] is so bad, it’s not even worth using.” – David Allison, obesity researcher
• “Many studies of nutritional epidemiology that try to link dietary exposures to disease outcomes are founded on really dodgy data,” – John Speakman, study co-author
• “We’ve got to move on,” he says. “We’ve got to try and use new technologies to do better.” – Gary Frost

Conclusions:

The study published in Nature Food underscores a critical problem with dietary research: widespread underreporting in self-reported data. While this issue is not new, the study uses DLW data to quantify the problem, raising questions about the validity of countless nutritional studies and calling for the development of more reliable data collection methods. While there is some pushback from researchers who think their work is valid despite the data issues, the overall message is that the field of nutritional epidemiology is in need of change.

FAQ: Reliability of Nutritional Studies

1. Why are there concerns about the reliability of nutritional studies that rely on self-reported dietary information? Many nutritional studies rely on questionnaires or food diaries where participants report what they eat. These self-reported methods are prone to inaccuracies because people often misremember what they ate, underestimate the quantity of food, or are reluctant to accurately report consumption of certain items. This leads to systematic underreporting, making the data unreliable.
2. What is the “doubly labeled water” (DLW) technique, and how does it help expose issues with self-reported dietary data? The DLW technique involves having participants drink water containing heavier isotopes of hydrogen and oxygen. By analyzing urine samples, scientists can measure how much energy a person has used based on how much of the oxygen isotope is exhaled as CO2. DLW provides a more objective measure of energy expenditure than dietary recall. Studies comparing DLW data to self-reported dietary intake have found a consistent pattern of people underreporting their food consumption, often by significant margins.
3. How did the recent study in Nature Food use DLW data to assess dietary surveys? Researchers used existing DLW data from over 6000 people to develop an equation that predicts an individual’s energy expenditure based on easily measurable characteristics such as age, sex, and body weight. This equation was then applied to the data from large nutritional surveys like NHANES and NDNS, to see if reported dietary intakes matched predicted energy expenditure. It found that a large percentage of survey records (over 50% in NHANES and 60% in NDNS) reported energy intakes lower than what the equation would predict, showing widespread underreporting.
4. What are the potential implications of these findings for existing nutritional research? The high levels of misreporting identified in major dietary surveys call into question thousands of studies that have used this data to link particular diets to health outcomes. If the underlying data is inaccurate, the conclusions drawn from those studies may also be unreliable. This could explain the frequently contradictory findings in nutritional research, where a specific food may be linked to a disease one time but not another.
5. Are all types of food reported with equal inaccuracy, or does it vary? The research suggests that the degree of misreporting varies depending on what types of food people are consuming. For example, people were found to underreport more when they ate more protein. This indicates that certain dietary patterns might be more susceptible to inaccurate reporting than others. This could make it harder to draw accurate conclusions for some types of diets than others.
6. Are self-reported dietary surveys completely useless, or can they still provide value? While acknowledging the inherent problems with self-reported data, some researchers defend dietary surveys as still the best data currently available. They emphasize that these surveys are conducted at a large scale and capture valuable information, and argue that, even with their limitations, they provide useful information. They also claim misreporting might not be severe enough to completely invalidate the results of well-conducted studies, especially when taken together with other sources of evidence.
7. What alternative methods are researchers exploring to improve the accuracy of dietary data? Researchers are exploring several alternatives to improve the accuracy of dietary research, including photographic food diaries, wearable cameras, and motion and audio sensors. Some researchers are also focusing on finding biomarkers in urine or blood samples that can indicate how much of a specific food a person has consumed. These methods seek to provide more objective data but are generally more expensive and difficult to implement at scale.
8. How can this new research be used to improve future nutritional studies, even before new methods become scalable? The equation derived from DLW data, and published in Nature Food, can be used by researchers to check their existing datasets and estimate the scale of misreporting. By understanding the degree of inaccuracy in the data, researchers can take this into account and interpret the results with more caution. This method can provide a step towards improving the credibility of nutritional research while they work on other methods to improve accuracy.