How well can multiple regression really “control for” covariates?

Question

We’re all familiar with observational studies that attempt to establish a causal link between a nonrandomized predictor X and an outcome by including every imaginable potential confounder in a multiple regression model. By thus “controlling for” all the confounders, the argument goes, we isolate the effect of the predictor of interest.

I’m developing a growing discomfort with this idea, based mostly on off-hand remarks made by various professors of my statistics classes. They fall into a few main categories:

1. You can only control for covariates that you think of and measure.
This is obvious, but I wonder if it is actually the most pernicious and insurmountable of all.

2. The approach has led to ugly mistakes in the past.

For example, Petitti & Freedman (2005) discuss how decades’ worth of statistically adjusted observational studies came to disastrously incorrect conclusions on the effect of hormone replacement therapy on heart disease risk. Later RCTs found nearly opposite effects.

3. The predictor-outcome relationship can behave strangely when you control for covariates.

Yu-Kang Tu, Gunnell, & Gilthorpe (2008) discuss some different manifestations, including Lord’s Paradox, Simpson’s Paradox, and suppressor variables.

4. It is difficult for a single model (multiple regression) to adequately adjust for covariates and simultaneously model the predictor-outcome relationship.

I’ve heard this given as a reason for the superiority of methods like propensity scores and stratification on confounders, but I'm not sure I really understand it.

5. The ANCOVA model requires the covariate and predictor of interest to be independent.

Of course, we adjust for confounders precisely BECAUSE they're correlated with the predictor of interest, so, it seems, the model will be unsuccessful in the exact instances when we want it the most. The argument goes that adjustment is only appropriate for noise-reduction in randomized trials. Miller & Chapman, 2001 give a great review.

So my questions are:

1. How serious are these problems and others I might not know of?
2. How afraid should I be when I see a study that "controls for everything"?

(I hope this question isn't venturing too far into discussion territory and happily invite any suggestions for improving it.)

EDIT: I added point 5 after finding a new reference.

Answer 1

There is a becoming widely accepted, non-statistical perhaps, answer to - what assumptions does one need to make to claim one has really controlled for the covariates.

That can be done with Judea Pearl's causal graphs and do calculus.

See http://ftp.cs.ucla.edu/pub/stat_ser/r402.pdf as well as other material on his website.

Now as statisticians we know that all models are false, and the real statistical question is are those identified assumption likely to be not too wrong so that our answer is approximately OK. Pearl is aware of this and does discuss it in his work but perhaps not explicitly and often enough to avoid flustrating many statisticians with his claim to have an answer (which I believe his does for what assumptions does one need to make?).

(Currently the ASA is offering a prize for teaching material to include these methods in statistical courses see here)