How well can multiple regression really "control for" covariates?

Question

We’re all familiar with observational studies that attempt to establish a causal link between a nonrandomized predictor X and an outcome by including every imaginable potential confounder in a multiple regression model. By thus “controlling for” all the confounders, the argument goes, we isolate the effect of the predictor of interest.

I’m developing a growing discomfort with this idea, based mostly on off-hand remarks made by various professors of my statistics classes. They fall into a few main categories:

1. You can only control for covariates that you think of and measure.
This is obvious, but I wonder if it is actually the most pernicious and insurmountable of all.

2. The approach has led to ugly mistakes in the past.

For example, Petitti & Freedman (2005) discuss how decades’ worth of statistically adjusted observational studies came to disastrously incorrect conclusions on the effect of hormone replacement therapy on heart disease risk. Later RCTs found nearly opposite effects.

3. The predictor-outcome relationship can behave strangely when you control for covariates.

Yu-Kang Tu, Gunnell, & Gilthorpe (2008) discuss some different manifestations, including Lord’s Paradox, Simpson’s Paradox, and suppressor variables.

4. It is difficult for a single model (multiple regression) to adequately adjust for covariates and simultaneously model the predictor-outcome relationship.

I’ve heard this given as a reason for the superiority of methods like propensity scores and stratification on confounders, but I'm not sure I really understand it.

5. The ANCOVA model requires the covariate and predictor of interest to be independent.

Of course, we adjust for confounders precisely BECAUSE they're correlated with the predictor of interest, so, it seems, the model will be unsuccessful in the exact instances when we want it the most. The argument goes that adjustment is only appropriate for noise-reduction in randomized trials. Miller & Chapman, 2001 give a great review.

So my questions are:

1. How serious are these problems and others I might not know of?
2. How afraid should I be when I see a study that "controls for everything"?

(I hope this question isn't venturing too far into discussion territory and happily invite any suggestions for improving it.)

EDIT: I added point 5 after finding a new reference.

Answer 1

There is a becoming widely accepted, non-statistical perhaps, answer to - what assumptions does one need to make to claim one has really controlled for the covariates.

That can be done with Judea Pearl's causal graphs and do calculus.

See http://ftp.cs.ucla.edu/pub/stat_ser/r402.pdf as well as other material on his website.

Now as statisticians we know that all models are false, and the real statistical question is are those identified assumption likely to be not too wrong so that our answer is approximately OK. Pearl is aware of this and does discuss it in his work but perhaps not explicitly and often enough to avoid flustrating many statisticians with his claim to have an answer (which I believe his does for what assumptions does one need to make?).

(Currently the ASA is offering a prize for teaching material to include these methods in statistical courses see here)

Answer 2

Answer to question 1:

• The magnitude of seriousness is best assessed in a contextual way (i.e., should consider all factors contributing to validity).
• The magnitude of seriousness should not be assessed in a categorical way. An example is the notion of a hierarchy of inference for study designs (e.g. case reports are lowest and RCTs are categorically highest). This type of scheme is frequently taught in medical schools as an easy heuristic to quickly identify high quality evidence. The problem with this type of thinking is that it is algorithmic and overly deterministic in reality the answer is itself overdetermined. When this happens, you can miss the ways in which poorly designed RCTs can yield worse results than a well designed observational study.
• See this easy to read review for a full discussion of the above points from the perspective of an epidemiologist (Rothman, 2014).

Answer to question 2:

• Be very afraid. To simply reiterate what others have already said and to quote (roughly) from Richard McElreath's elegant introductory text on critical thinking in statistical modeling:

  "...all models are false, but some are useful..."