What are the case studies in public health policy research where unreliable/confounded/invalid studies or models were misused?

Question

I am drafting a literature review on a current public health issue where data are confounded:

What are common historical case-studies that are used in public health/epidemiology education where invalid or confounded relationships or inferences were intentionally or erroneously employed in public health policy and legislation?

The automobile fatality surge of the 1960s and subsequent evidence-based, government-led study which determined seatbelts and eventually airbags should be required by law is a great example of HOW public health policy should be driven by statistically powerful inferences and models.

I am looking more for examples of cases of the opposite type (bad science to make policy hastily). However, if nothing else I would like to learn of more cases similar to the previous example of powerful studies for successful public health benefit.

I want to use these as examples to demonstrate how evidence-based statistical public health research is important to the making of policy.

Answer 1

I think the best example of this may likely be the controversy around hormone replacement therapy and cardiovascular risk - large cohort epidemiological studies seem to suggest a protective effect and health policy and physician recommendations were made on this information.

Follow-up RCTs then seem to show that there's actually an increased risk of myocardial infarction in women placed on HRT.

This goes back and forth for a bit, and has been used as one of the canonical cases to attack epidemiology as a field, but a recent re-analysis by Hernan seems to propose that the two studies actually don't have discordant results if you make sure you ask the same question.

Answer 2

A really interesting example I personally like is taken from the book Freakonomics by Steven D. Levitt and Stephen J. Dubner. There is a chapter in the book that discusses correlation vs. causality. Correlation between two statistical variables does not necessarily imply that these variables are statistically dependent, but a mistake along these lines was made by experts. Quoting from the book:

"A tricky beast, Polio was extremely difficult for researchers to pin down. They couldn’t figure out how it was passed or when/how it expressed itself. We have a tendency to remember this time as one in which Polio was ‘epidemic’ when, in fact, it was not affecting large swaths of the population (compared with the more common measles, for example). The reason it was seen as epidemic was because it was so frightening.

What researchers DID manage to determine in their studies was that Polio infection rates went UP in the Summer. They also saw that ICE CREAM CONSUMPTION went up in the Summer. And so they concluded that ice cream consumption led to Polio and for a time ice cream was demonized. "

Answer 3

In his paper, "Statistical Models and Shoe Leather" (1991), David Freedman presents some cautionary tales in epidemiological studies. He offers Snow's analysis of cholera in London as a success, not due to statistical modeling, but rather due to diligent data collection. Here's the abstract:

Regression models have been used in the social sciences at least since 1899, when Yule published a paper on the causes of pauperism. Regression models are now used to make causal arguments in a wide variety of applications, and it is perhaps time to evaluate the results. No definitive answers can be given, but this paper takes a rather negative view. Snow's work on cholera is presented as a success story for scientific reasoning based on nonexperimental data. Failure stories are also discussed, and comparisons may provide some insight. In particular, this paper suggests that statistical technique can seldom be an adequate substitute for good design, relevant data, and testing predictions against reality in a variety of settings.

Sociological Methodology. 21: 291-313.

Answer 4

The case of high-dose chemotherapy with bone-marrow-transplant rescue as treatment for advanced breast cancer in the 1990's is one such instance. A series of low-quality studies were used to push through legislation mandating health insurer coverage in some states. When the large randomized trials were completed, there was no measurable benefit.

http://www.gao.gov/products/HEHS-96-83