Are there any well-known statistical studies that were originally published and thought to be valid, but later had to be thrown out due to a confounding variable that wasn't taken into account? I'm looking for something easy to understand that could be explained to and appreciated by a quantitative literacy class that has zero pre-requisites.
$\begingroup$
$\endgroup$
4
-
7$\begingroup$ What is the difference between "quantitative literacy" and "numeracy" as a state of mind? $\endgroup$– HenryCommented Oct 23, 2019 at 23:35
-
1$\begingroup$ The Stanford Marshmallow Experiment is the first one that came to my mind. $\endgroup$– Apollys supports MonicaCommented Oct 25, 2019 at 21:23
-
1$\begingroup$ I think of all the public opinion polls that assume no correlation between a person’s opinion and their willingness to answer questions from strangers (or their willingness to tolerate a phone call interrupting whatever). $\endgroup$– WGroleauCommented Oct 26, 2019 at 15:04
-
1$\begingroup$ It's just the preferred terminology. But I guess really to me numeracy means having a great grasp on number sense itself, whereas in our QL class we try to teach a smattering of lots of things - but probably none of them too well (such as statistics and logic). $\endgroup$– NathanLiteCommented Oct 28, 2019 at 14:52
Add a comment
|
12 Answers
$\begingroup$
$\endgroup$
7
Coffee Drinking & Lung Cancer
My favorite example is that supposedly, "coffee drinkers have a greater risk of lung cancer", despite most coffee drinkers... well... drinking coffee, rather than inhaling it.
There have been various studies about this, but the consensus remains that studies with this conclusion usually just have a larger proportion of smoking coffee drinkers, than non-smoking coffee drinkers. In other words, the effect of smoking confounds the effect of coffee consumption, if not included in the model. The most recent article on this I could find is a meta analysis by Vania Galarraga and Paolo Boffetta (2016).$^\dagger$
The Obesity Paradox
Another example that plagues clinical research, is the claim that obesity can be beneficial for certain diseases. Specifically, many articles, still to this day (just do a quick search for obesity paradox on pubmed and be amazed), claim the following:
- While a higher BMI increases the risk of diabetes, cardiovascular disease and certain types of cancer, once a patient already has the disease, a higher BMI is associated with lower rates of major adversarial events or death.
Why does this happen? Obesity is defined as excess fat negatively affecting health, yet we classify obesity based on BMI. BMI is just calculated as:
$$\text{BMI} = \frac{\text{weight in kg}}{(\text{height in m})^2},$$
so the most direct way to combat obesity is through weight loss (or by growing taller somehow).
Regimes that focus on loss of weight rather than fat, tend to result in a proportionally large loss of muscle. This is likely what causes lower BMI to be associated with a higher rate of major adversarial events.
Because many studies do not include measures of body fat (percentage), but only BMI as a proxy, the amount of body fat confounds the effect of BMI on health.
A nice review of this phenomenon was written by Steven G. Chrysant (2018).$^\ddagger$ He ends with:
[B]ased on the recent evidence, the obesity paradox is a misnomer and could convey the wrong message to the general public that obesity is not bad.
Followed by:
Journals [should] no longer accept articles about the 'obesity paradox'.
$\dagger$: Vania Galarraga and Paolo Boffetta (2016): Coffee Drinking and Risk of Lung Cancer—A Meta-Analysis. Cancer Epidemiol Biomarkers Prev June 1 2016 (25) (6) 951-957; DOI: 10.1158/1055-9965.EPI-15-0727
$\ddagger$: Steven G. Chrysant (2018): Obesity is bad regardless of the obesity paradox for hypertension and heart disease. J Clin Hypertens (Greenwich). 2018 May;20(5):842-846. doi: 10.1111/jch.13281. Epub 2018 Apr 17.
Examples of (poor) studies claiming to have demonstrated the obesity paradox:
- McAuley et al. (2018): Exercise Capacity and the Obesity Paradox in Heart Failure: The FIT (Henry Ford Exercise Testing) Project
- Weatherald et al. (2018): The association between body mass index and obesity with survival in pulmonary arterial hypertension
- Patel et al. (2018): The obestiy paradox: the protective effect of obesity on right ventricular function using echocardiographic strain imaging in patients with pulmonary hypertension
Articles refuting the obesity paradox as a mere confounding effect of body fat:
- Lin et al. (2017): Impact of Misclassification of Obesity by Body Mass Index on Mortality in Patients With CKD
- Leggio et al. (2018): High body mass index, healthy metabolic profile and low visceral adipose tissue: The paradox is to call it obesity again
- Medina-Inojosa et al. (2018): Association Between Adiposity and Lean Mass With Long-Term Cardiovascular Events in Patients With Coronary Artery Disease: No Paradox
- Flegal & Ioannidis (2018): The Obesity Paradox: A Misleading Term That Should Be Abandoned
Articles about the obesity paradox in cancer:
- Cespedes et al. (2018): The Obesity Paradox in Cancer: How Important Is Muscle?
- Caan et al. (2018): The Importance of Body Composition in Explaining the Overweight Paradox in Cancer-Counterpoint
-
9$\begingroup$ Another example I recall from our classes was the correlation of the amount of fire engines involved in fighting a fire with the damage of the fire. The confounding variable there was that the bigger the fire, the bigger the damages, but also the more engines were needed to fight the fire. I can't recall how serious those studies were or if it was just some toy example. $\endgroup$– JADCommented Oct 24, 2019 at 9:33
-
1$\begingroup$ On the obesity paradox --- older subjects with higher BMI suffer fewer broken bones (hips etc) through falls it seems. Do they fall less? Are they 'cushioned' by fat (the Paradox explanation)? Have they got less athropied muscles and other health problems thus fewer falls? Etc. $\endgroup$ Commented Oct 24, 2019 at 13:41
-
1$\begingroup$ To clarify, is the explanation of the obesity paradox that some people who are diagnosed with a disease try to lose weight, and consequently lose muscle, which has a negative effect, while others who do not try to lose weight, and therefore remain obese, do not suffer this effect? $\endgroup$ Commented Oct 24, 2019 at 17:08
-
1$\begingroup$ On the obesity paradox, consider that many cancers cause the patient to lose a lot of weight, and "unexplained weight loss" is one of worrying symptoms that should lead a person to see a doctor - I'm convinced that this also has something do to with the paradox. $\endgroup$ Commented Oct 24, 2019 at 21:05
-
$\begingroup$ I think I remember reading a "joke" study that determined that in football being ahead in the 3rd quarter was highly correlated with winning the game. The "takeaway" for coaches was to win more games, they should score more points than the other team. $\endgroup$– emoryCommented Oct 25, 2019 at 11:35
$\begingroup$
$\endgroup$
1
You might want to introduce Simpson's Paradox.
The first example that page is the UC Berkeley gender bias case where it was thought that there was gender bias (towards males) in admissions when looking at overall acceptance rates, but this was eliminated or reversed when investigated by department. The confounding variable of department picked up on a gender difference in applying to more competetive departments.
-
$\begingroup$ Another takeaway from that should be to investigate if this applying to more competitive departments (suppose for women) is a more general phenomenon? Anybody have references/links about that? $\endgroup$ Commented Dec 1, 2019 at 5:28
$\begingroup$
$\endgroup$
3
Power Lines and Cancer
After an initial study finding a link between living next to high-voltage transmission lines and cancer, follow-up studies found that when you include income in the model the effect of the power lines goes away.
Living next to power lines is a moderately accurate predictor of low household income / wealth. Put bluntly, there aren't as many fancy mansions next to transmission lines as elsewhere.
There is correlation between poverty and cancer. When comparisons were made between households on similar income brackets close to and far away from transmission lines, the effect of transmission lines disappeared.
In this case, the confounding variables were household wealth and distance to the nearest high voltage line.
-
$\begingroup$ As I recall the author of these studies was later shown to have outright falsified his data, the results were discredited and he is now prevented from doing research in this field. $\endgroup$– mehCommented Oct 29, 2019 at 19:21
-
-
$\begingroup$ @aginensky The original study found correlation (as Jason said), but incorrectly claimed causation without sufficient data or evidence. The link does reference another researcher who did falsify a study in 1992 though (who was later fired in 1999 when the falsification was discovered). $\endgroup$– GrahamCommented Dec 2, 2019 at 11:59
$\begingroup$
$\endgroup$
Consider the following examples. I am not sure they are necessarily very famous but they help to demonstrate the potential negative effects of confounding variables.
Say one is studying the relation between birth order (1st child, 2nd child, etc.) and the presence of Down Syndrome in the child. In this scenario, maternal age would be a confounding variable:
Higher maternal age is directly associated with Down Syndrome in the child
Higher maternal age is directly associated with Down Syndrome, regardless of birth order (a mother having her 1st vs 3rd child at age 50 confers the same risk)
Maternal age is directly associated with birth order (the 2nd child, except in the case of twins, is born when the mother is older than she was for the birth of the 1st child)
Maternal age is not a consequence of birth order (having a 2nd child does not change the mother's age)
More examples
In risk assessments, factors such as age, gender, and educational levels often affect health status and so should be controlled. Beyond these factors, researchers may not consider or have access to data on other causal factors. An example is the study of smoking tobacco on human health. Smoking, drinking alcohol, and diet are lifestyle activities that are related. A risk assessment that looks at the effects of smoking but does not control for alcohol consumption or diet may overestimate the risk of smoking (Tjønneland, Grønbaek, Stripp, & Overvad, 1999). Smoking and confounding are reviewed in occupational risk assessments such as the safety of coal mining (Axelson, 1989). When there is not a large sample population of non-smokers or non-drinkers in a particular occupation, the risk assessment may be biased towards finding a negative effect on health.
References: https://en.wikipedia.org/wiki/Confounding
Tjønneland, A., Grønbaek, M., Stripp, C., & Overvad, K. (1999). Wine intake and diet in a random sample of 48763 Danish men and women. The American Journal of Clinical Nutrition, 69(1), 49-54.
Axelson, O. (1989). Confounding from smoking in occupational epidemiology. British Journal of Industrial Medicine, 46(8), 505-507.
$\begingroup$
$\endgroup$
2
There was one about diet that looked at diet in different countries and concluded that meat caused all sorts of problems (e.g. heart disease), but failed to account for the average lifespan in each country: The countries that ate very little meat also had lower life expectancies and the problems that meat "caused" were ones that were linked to age.
I don't have citations for this - I read about it about 25 years ago - but maybe someone will remember or maybe you can find it.
-
1$\begingroup$ Are you sure it wasn’t fat rather than meat? The infamous “seven countries study” that started the don’t-eat-fat myth not only ignored a few variables but strangely omitted data from many countries. $\endgroup$– WGroleauCommented Oct 26, 2019 at 15:01
-
2$\begingroup$ I'm sort of sure it was meat. There could easily have been multiple studies. Diet is notorious for bad statistics. $\endgroup$ Commented Oct 26, 2019 at 18:54
$\begingroup$
$\endgroup$
2
I'm not sure it entirely counts as a confounding variable so much as confounding situations, but animals' abilities to find their way through a maze may qualify.
As described in this ScienceDirect summary, studies of rats (or other animals) in mazes were popular for a large part of the 20th century, and continue today to some extent. One possible purpose is to study the subject's ability to remember a maze which it has previously run; another popular purpose is to study any bias in the subject's choices of whether to turn left or right at junctions, in a maze which the subject has not previously run.
It should be immediately clear that if the subject has forgotten the maze, then any inherent bias in choice of route will be a confounding factor. If the "right" direction coincides with the subject's bias, then they could find their way in spite of not remembering the route.
In addition to this, studies found various other confounding features exist which might not have been considered. The height of walls and width of passages are factors, for example. And if another subject has previously navigated the maze, subjects which rely strongly on their sense of smell (mice and dogs, for instance) may find their way simply by tracking the previous subject's scent. Even the construction of the maze may be an issue - animals tend to be less happy to run over "hollow-sounding" floors.
Many animal maze studies ended up finding confounding factors instead of the intended study results. More disturbingly, according to Richard Feynmann, the studies reporting these confounding factors were not picked up by researchers at the time. As a result we simply don't know if any animal maze studies carried out around this time have any validity whatsoever. That's decades worth of high-end research at the finest universities around the world, by the finest psychologists and animal behaviourists, and every last shred of work had to at best be taken with a very large spoon of salt. Later researchers had to go back and duplicate all this work, to find out what was actually valid and what wasn't repeatable.
-
1$\begingroup$ If I read your link correctly, it appears that Richard Feynmann, who is not a researcher in ethology, animal behaviour or psychology commented in a talk (not an academic conference or publication of any kind) on some unidentified unpublished “study” that may or may not have showed what he says it showed and then concluded that researchers in the field do not care about this because they do not talk about “Mr. Young”. And that's supposed to be an example of scientific rigor or evidence that researchers in the field do not care about confounding factors in general? $\endgroup$– GalaCommented Oct 26, 2019 at 10:56
-
1$\begingroup$ @Gala Not that they did not care, but that they did not know. Feynman's point wasn't about scientific rigor from individual scientists - he had no doubt they did the best they could. His point was that the scientific community as a whole was substandard at passing on this important information so that other people don't fall into the same trap. He was using this as a nice hook to hang his lesson on, but there are so many examples of this - Mendel, for instance. $\endgroup$– GrahamCommented Oct 31, 2019 at 0:59
$\begingroup$
$\endgroup$
There was a great study of mobile phone use and brain cancer. Most people with a lateral brain cancer, when asked which hand they hold their phone in, answer the diseased side. This seemed to show that phone use caused cancer.
However, maybe the answers are informed by hindsight. Someone thought of a great test for this. The sample was big enough to include some people with two cancers. So you could ask, does the declared side of phone use influence the risk of a cancer on the other side of the brain? It was actually protective, thus showing the hindsight bias in the original result.
Sorry, I don't have the reference.
$\begingroup$
$\endgroup$
'Statistics' by Freedman, Purvis et al. has a number of examples in the first couple of chapters. My personal favorite is that ice cream causes polio. The confounding variable is that they are both prevalent in the summertime when young children are out, about, and spreading polio. The book is "Statistics (Fourth Edition) 4th Edition, Kindle Edition- by David Freedman (Author), Robert Pisani (Author), Roger Purves (Author)"
$\begingroup$
$\endgroup$
This is not a study, but a gallery of spurious correlations that could be appreciated by a quantitative literacy class. The downside of this is the lack of an explanation (aside from chance).
$\begingroup$
$\endgroup$
See: Subversive Subjects: Rule-Breaking and Deception in Clinical Trials https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4520402/
$\begingroup$
$\endgroup$
Hormone replacement Therapy and heart disease?
The benefits were determined by observation, and essentially it appears that the people who chose to do hrt had higher socioeconomic status, healthier lifestyle etc
(So one could argue on confounding Vs observational study)
$\begingroup$
$\endgroup$
There are lots of good examples in Howard Weiner's books. In particular, Chapter 1 "The most dangerous equation" in "How to understand, communicate and control uncertainty through graphical display"
Examples include:
The small schools movement.
People noticed that some small schools had better performance than large schools so spent money to reduce school size. It turned out that some small schools also had worse performance than large schools. It was largely an artefact of extreme outcomes showing up in small samples.Kidney cancer rates (This example is also used Daniel Kahneman's "Thinking Fast and Slow", see the start of Chapter 10). Lowest kidney cancer rates in rural, sparsely populated counties. These low rates have to be because of the the clean living rural life style. But wait, counties with the highest incidence of kidney cancer are also rural and sparsely populated. This has to be because of the lack of access to good medical care and too much drinking. Of course, the extreme are actually an artefact of the small populations.