Statistics and causal inference?

In his 1984 paper "Statistics and Causal Inference", Paul Holland raised one of the most fundamental questions in statistics:

What can a statistical model say about causation?

This led to his motto:

NO CAUSATION WITHOUT MANIPULATION

which emphasized the importance of restrictions around experiments that consider causation. Andrew Gelman makes a similar point:

"To find out what happens when you change something, it is necessary to change it."...There are things you learn from perturbing a system that you'll never find out from any amount of passive observation.

His ideas are summarized in this article.

What considerations should be made when making a causal inference from a statistical model?

-
great question: see also this related question on correlation and causation stats.stackexchange.com/questions/534/… – Jeromy Anglim Sep 1 '10 at 5:14
To much to say. But you can read Pearl's book "Causality" (2002, but newer 2nd edition), or Hernan and Robins' book "Causal Inference" (2015, free electronic draft online if you search). – user52423 Jul 20 '14 at 19:42

3 Answers

This is a broad question, but given the Box, Hunter and Hunter quote is true I think what it comes down to is

1) The quality of the experimental design - randomization, sample sizes, control of confounders,...

2) The quality of the implementation of the design. - adherance to protocol, measurement error, data handling, ...

3) The quality of the model to accurately reflect the design. - blocking structures are accurately represented, proper degrees of freedom are associated with effects, estimators are unbiased, ...

At the risk of stating the obvious I'll try to hit on the key points of each:

(1) is a large sub-field of statistics, but in it's most basic form I think it comes down to the fact that when making causal inference we ideally start with identical units that are monitored in identical environments other than being assigned to a treatment. Any systematic differences between groups after assigment are then logically attributable to the treatment (we can infer cause). But, the world isn't that nice and units differ prior to treatment and evironments during experiments are not perfectly controlled. So we "control what we can and randomize what we can't", which helps to insure that there won't be systematic bias due to the confounders that we controlled or randomized. One problem is that experiments tend to be difficult (to impossible) and expensive and a large variety of designs have been developed to efficiently extract as much information as possible in as carefully controlled a setting as possible, given the costs. Some of these are quite rigorous (e.g. in medicine the double-blind, randomized, placebo-controlled trial) and others less so (e.g. various forms of 'quasi-experiments').

(2) is also a big issue and one that statisticians generally don't think about...though we should. In applied statistical work I can recall incidences where 'effects' found in the data were spurious results of inconsistency of data collection or handling. I also wonder how often information on true causal effects of interest is lost due to these issues (I believe students in the applied sciences generally have little-to-no training about ways that data can become corrupted - but I'm getting off topic here...)

(3) is another large technical subject, and another necessary step in objective causal inference. To a certain degree this is taken care of because the design crowd develop designs and models together (since inference from a model is the goal, the attributes of the estimators drive design). But this only gets us so far because in the 'real world' we end up analysing experimental data from non-textbook designs and then we have to think hard about things like the appropriate controls and how they should enter the model and what associated degrees of freedom should be and whether assumptions are met if if not how to adjust of violations and how robust the estimators are to any remaining violations and...

Anyway, hopefully some of the above helps in thinking about considerations in making causal inference from a model. Did I forget anything big?

-
A huge plus one for point 2. Other than going through human subjects protection training, I've never received the tiniest bit of training on data collection and storage. Getting the data collection right is vastly more important than the analysis. – Matt Parker Sep 3 '10 at 18:54
I'd love to answer too, but I'm afraid there's nothing left to add to what Kingsford said. – Joris Meys Sep 8 '10 at 21:43

In addition to the excellent answer above, there is a statistical method that can get you closer to demonstrating causality. It is Granger Causality that demonstrates that one independent variable occurring before a dependent variable has a causal effect or not. I introduce this method in an easy to follow presentation at the following link:

http://www.slideshare.net/gaetanlion/granger-causality-presentation

I also apply this method to testing competing macroeconomic theories: http://www.slideshare.net/gaetanlion/economic-theory-testing-presentation

Be aware that this method is not perfect. It just confirms that certain events occur before others and that those events appear to have a consistent directional relationship. This seems to entail true causality but it is not always the case. The rooster morning call does not cause the sun to rise.

-
-

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

+1 the classic. – Henrik Sep 18 '10 at 8:21