We’re all familiar with observational studies that attempt to establish a causal link between a nonrandomized predictor X and an outcome by including every imaginable potential confounder in a multiple regression model. By thus “controlling for” all the confounders, the argument goes, we isolate the effect of the predictor of interest.

I’m developing a growing discomfort with this idea, based mostly on off-hand remarks made by various professors of my statistics classes. They fall into a few main categories:

1. You can only control for covariates that you think of and measure.
This is obvious, but I wonder if it is actually the most pernicious and insurmountable of all.

2. The approach has led to ugly mistakes in the past.

For example, Petitti & Freedman (2005) discuss how decades’ worth of statistically adjusted observational studies came to disastrously incorrect conclusions on the effect of hormone replacement therapy on heart disease risk. Later RCTs found nearly opposite effects.

3. The predictor-outcome relationship can behave strangely when you control for covariates.

Yu-Kang Tu, Gunnell, & Gilthorpe (2008) discuss some different manifestations, including Lord’s Paradox, Simpson’s Paradox, and suppressor variables.

4. It is difficult for a single model (multiple regression) to adequately adjust for covariates and simultaneously model the predictor-outcome relationship.

I’ve heard this given as a reason for the superiority of methods like propensity scores and stratification on confounders, but I'm not sure I really understand it.

5. The ANCOVA model requires the covariate and predictor of interest to be independent.

Of course, we adjust for confounders precisely BECAUSE they're correlated with the predictor of interest, so, it seems, the model will be unsuccessful in the exact instances when we want it the most. The argument goes that adjustment is only appropriate for noise-reduction in randomized trials. Miller & Chapman, 2001 give a great review.

So my questions are:

  1. How serious are these problems and others I might not know of?
  2. How afraid should I be when I see a study that "controls for everything"?

(I hope this question isn't venturing too far into discussion territory and happily invite any suggestions for improving it.)

EDIT: I added point 5 after finding a new reference.

  • 1
    $\begingroup$ For question 2, I think 'controls for everything' is a more general issue of specification. I have trouble thinking of a situation where a parametric model is correctly specified. That being said, a model simplifies reality, and that is where the art of this type of study lies. The researcher has to decide what is and is not important in the model. $\endgroup$
    – kirk
    Nov 1 '12 at 13:59
  • 4
    $\begingroup$ With this question you've made me a fan. $\endgroup$
    – rolando2
    Nov 1 '12 at 16:40
  • 1
    $\begingroup$ I think this raises some very good points; but I think the answers are outside the strictly statistical field. Thus, any statistical result is more valuable if it 1) Is replicated 2) Is substantively viable etc. Also see the MAGIC criteria and the general argument Abelson makes. $\endgroup$
    – Peter Flom
    Nov 2 '12 at 12:01
  • 1
    $\begingroup$ Point #5 is absolutely false. The Miller & Chapman paper is completely wrong, full stop. $\endgroup$ Apr 10 '18 at 4:11
  • 1
    $\begingroup$ @half-pass Not sure what else to say about it other than that the central claim of the paper -- i.e., that the focal predictor X and covariate C must be uncorrelated -- is just not true. Notice that ANCOVA is just a regression model, so this same line of reasoning would apparently invalidate nearly all real-world uses of multiple regression as well! I had some Twitter discussion about this awful paper several months ago: twitter.com/CookieSci/status/902298218494644228 $\endgroup$ Apr 10 '18 at 14:19

There is a becoming widely accepted, non-statistical perhaps, answer to - what assumptions does one need to make to claim one has really controlled for the covariates.

That can be done with Judea Pearl's causal graphs and do calculus.

See http://ftp.cs.ucla.edu/pub/stat_ser/r402.pdf as well as other material on his website.

Now as statisticians we know that all models are false, and the real statistical question is are those identified assumption likely to be not too wrong so that our answer is approximately OK. Pearl is aware of this and does discuss it in his work but perhaps not explicitly and often enough to avoid flustrating many statisticians with his claim to have an answer (which I believe his does for what assumptions does one need to make?).

(Currently the ASA is offering a prize for teaching material to include these methods in statistical courses see here)

  • $\begingroup$ Great reference to an elegant graphical representation, thank you. $\endgroup$
    – half-pass
    Nov 19 '12 at 17:31

Answer to question 1:

  • The magnitude of seriousness is best assessed in a contextual way (i.e., should consider all factors contributing to validity).
  • The magnitude of seriousness should not be assessed in a categorical way. An example is the notion of a hierarchy of inference for study designs (e.g. case reports are lowest and RCTs are categorically highest). This type of scheme is frequently taught in medical schools as an easy heuristic to quickly identify high quality evidence. The problem with this type of thinking is that it is algorithmic and overly deterministic in reality the answer is itself overdetermined. When this happens, you can miss the ways in which poorly designed RCTs can yield worse results than a well designed observational study.
  • See this easy to read review for a full discussion of the above points from the perspective of an epidemiologist (Rothman, 2014).

Answer to question 2:


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.