Statistics and causal inference?

Question

In his 1984 paper "Statistics and Causal Inference", Paul Holland raised one of the most fundamental questions in statistics:

What can a statistical model say about causation?

This led to his motto:

NO CAUSATION WITHOUT MANIPULATION

which emphasized the importance of restrictions around experiments that consider causation. Andrew Gelman makes a similar point:

"To find out what happens when you change something, it is necessary to change it."...There are things you learn from perturbing a system that you'll never find out from any amount of passive observation.

His ideas are summarized in this article.

What considerations should be made when making a causal inference from a statistical model?

Answer 1

This is a broad question, but given the Box, Hunter and Hunter quote is true I think what it comes down to is

1. The quality of the experimental design:

   • randomization, sample sizes, control of confounders,...

2. The quality of the implementation of the design:

   • adherance to protocol, measurement error, data handling, ...

3. The quality of the model to accurately reflect the design:

   • blocking structures are accurately represented, proper degrees of freedom are associated with effects, estimators are unbiased, ...

At the risk of stating the obvious I'll try to hit on the key points of each:

1. is a large sub-field of statistics, but in it's most basic form I think it comes down to the fact that when making causal inference we ideally start with identical units that are monitored in identical environments other than being assigned to a treatment. Any systematic differences between groups after assigment are then logically attributable to the treatment (we can infer cause). But, the world isn't that nice and units differ prior to treatment and evironments during experiments are not perfectly controlled. So we "control what we can and randomize what we can't", which helps to insure that there won't be systematic bias due to the confounders that we controlled or randomized. One problem is that experiments tend to be difficult (to impossible) and expensive and a large variety of designs have been developed to efficiently extract as much information as possible in as carefully controlled a setting as possible, given the costs. Some of these are quite rigorous (e.g. in medicine the double-blind, randomized, placebo-controlled trial) and others less so (e.g. various forms of 'quasi-experiments').

2. is also a big issue and one that statisticians generally don't think about...though we should. In applied statistical work I can recall incidences where 'effects' found in the data were spurious results of inconsistency of data collection or handling. I also wonder how often information on true causal effects of interest is lost due to these issues (I believe students in the applied sciences generally have little-to-no training about ways that data can become corrupted - but I'm getting off topic here...)

3. is another large technical subject, and another necessary step in objective causal inference. To a certain degree this is taken care of because the design crowd develop designs and models together (since inference from a model is the goal, the attributes of the estimators drive design). But this only gets us so far because in the 'real world' we end up analysing experimental data from non-textbook designs and then we have to think hard about things like the appropriate controls and how they should enter the model and what associated degrees of freedom should be and whether assumptions are met if if not how to adjust of violations and how robust the estimators are to any remaining violations and...

Anyway, hopefully some of the above helps in thinking about considerations in making causal inference from a model. Did I forget anything big?

Answer 2

What can a statistical model say about causation? What considerations should be made when making a causal inference from a statistical model?

The first thing to make clear is that you can't make causal inference from a purely statistical model. No statistical model can say anything about causation without causal assumptions. That is, to make causal inference you need a causal model.

Even in something considered as the gold standard, such as Randomized Control Trials (RCTs), you need to make causal assumptions to proceed. Let me make this clear. For example, suppose $Z$ is the randomization procedure, $X$ the treatment of interest and $Y$ the outcome of interest. When assuming a perfect RCT, this is what you are assuming:

In this case $P(Y|do(X)) = P(Y|X)$ so things are working well. However, suppose you have imperfect compliance resulting in a confounded relation between $X$ and $Y$. Then, now, your RCT looks like this:

You can still do an intent to treat analysis. But if you want to estimate the actual effect of $X$ things are not simple anymore. This is an instrumental variable setting, and you might be able to bound or even point identify the effect if you make some parametric assumptions.

This can get even more complicated. You may have measurement error problems, subjects might drop the study or not follow instructions, among other issues. You will need to make assumptions about how those things are related to procede with inference. With "purely" observational data this can be more problematic, because usually researchers will not have a good idea of the data generating process.

Hence, to draw causal inferences from models you need to judge not only its statistical assumptions, but most importantly its causal assumptions. Here are some common threats to causal analysis:

• Incomplete/imprecise data
• Target causal quantity of interest not well defined (What is the causal effect that you want to identify? What is the target population?)
• Confounding (unobserved confounders)
• Selection bias (self-selection, truncated samples)
• Measurement error (that can induce confounding, not only noise)
• Misspecification (e.g., wrong functional form)
• External validity problems (wrong inference to target population)

Sometimes the claim of absence of these problems (or the claim to have addressed these problems) can be backed up by the design of the study itself. That's why experimental data is usually more credible. Sometimes, however, people will assume away these problems either with theory or for convenience. If the theory is soft (like in the social sciences) it will be harder to take the conclusions at face value.

Anytime you think there's an assumption that can't be backed up, you should assess how sensitive the conclusions are to plausible violations of those assumptions --- this is usually called sensitivity analysis.

Answer 3

In addition to the excellent answer above, there is a statistical method that can get you closer to demonstrating causality. It is Granger Causality that demonstrates that one independent variable occurring before a dependent variable has a causal effect or not. I introduce this method in an easy to follow presentation at the following link:

http://www.slideshare.net/gaetanlion/granger-causality-presentation

I also apply this method to testing competing macroeconomic theories: http://www.slideshare.net/gaetanlion/economic-theory-testing-presentation

Be aware that this method is not perfect. It just confirms that certain events occur before others and that those events appear to have a consistent directional relationship. This seems to entail true causality but it is not always the case. The rooster morning call does not cause the sun to rise.