We’re all familiar with observational studies that attempt to establish a causal link between a nonrandomized predictor X and an outcome by including every imaginable potential confounder in a multiple regression model. By thus “controlling for” all the confounders, the argument goes, we isolate the effect of the predictor of interest.
I’m developing a growing discomfort with this idea, based mostly on off-hand remarks made by various professors of my statistics classes. They fall into a few main categories:
1. You can only control for covariates that you think of and measure.
This is obvious, but I wonder if it is actually the most pernicious and insurmountable of all.
2. The approach has led to ugly mistakes in the past.
For example, Petitti & Freedman (2005) discuss how decades’ worth of statistically adjusted observational studies came to disastrously incorrect conclusions on the effect of hormone replacement therapy on heart disease risk. Later RCTs found nearly opposite effects.
3. The predictor-outcome relationship can behave strangely when you control for covariates.
Yu-Kang Tu, Gunnell, & Gilthorpe (2008) discuss some different manifestations, including Lord’s Paradox, Simpson’s Paradox, and suppressor variables.
4. It is difficult for a single model (multiple regression) to adequately adjust for covariates and simultaneously model the predictor-outcome relationship.
I’ve heard this given as a reason for the superiority of methods like propensity scores and stratification on confounders, but I'm not sure I really understand it.
5. The ANCOVA model requires the covariate and predictor of interest to be independent.
Of course, we adjust for confounders precisely BECAUSE they're correlated with the predictor of interest, so, it seems, the model will be unsuccessful in the exact instances when we want it the most. The argument goes that adjustment is only appropriate for noise-reduction in randomized trials. Miller & Chapman, 2001 give a great review.
So my questions are:
- How serious are these problems and others I might not know of?
- How afraid should I be when I see a study that "controls for everything"?
(I hope this question isn't venturing too far into discussion territory and happily invite any suggestions for improving it.)
EDIT: I added point 5 after finding a new reference.