Interview question: If correlation doesn't imply causation, how do you detect causation?

Question

I got this question:

If correlation doesn't imply causation, how do you detect causation?

in an interview.

My answer was: You do some form of A/B testing. The interviewer kept prodding me for another approach but I couldn't think of any, and he wouldn't tell me whether my initial response was correct or not.

Are there any other approaches? And was my response correct?

Answer 1

There are a few ways around this. You are right that A/B testing is one of these. The economics Nobel this year was awarded for the pioneering of field experiments in the study of policies against poverty which do exactly this.

Otherwise, you could go about one of the following alternatives:

1. Selection on observables. Probably the most popular approach. You assume that conditional on some control variables, treatment assignment is random. In what is called the potential outcomes framework, under a binary treatment you could state this assumption as $Y_i(1), Y_i(0) \perp T_i \mid X_i$ where $T_i\in\{0,1\}$, $Y_i(t)$ are unit $i$’s outcome under treatment status $t$, and $X_i$ is a vector of $i$’s characteristics. The ideal way to achieve this is to randomize $T_i$. But other approaches that rely on this assumption are matching (including ML methods such as causal trees), inverse probability weighting, and the more ubiquitous method of adding $X_i$ as additional covariates in a linear regression. Computer science has gifted us with the theory of “directed acyclic graphs” for causal inference which help us think about what are good and what are bad variables to include in $X_i$.
2. Regression discontinuity designs. This method is very popular because it offers credible interpretation of results as causal. To illustrate the idea, take the example of a spatial discontinuity. Suppose there was an earthquake and kids in a certain zone were mandated to not go to school for 3 months. Kids just outside the border had no disruption in going to school. So you can compare kids just inside the zone to those just outside, and plausibly the only thing that will be different between them is school attendance. You can then regress their subsequent years of schooling, college attendance, etc., on which side of the border they lived, and get the causal effects of school attendance. Note that how to choose the right window around the discontinuity and implement the RD estimator is a subtle question and there is a literature behind this (see @olooney’s comment to this answer).
3. Instrumental variables. This is similar to regression discontinuity but usually much more difficult to defend. An instrument is a variable that you believe is only correlated with the outcome through the treatment status (that is, through the variable whose effect you want to measure). If this is the case, you can use something called two-stage least squares to estimate the causal effect. This genre has a small library’s worth of research on how things can go wrong if the assumptions fail, and even if they do not fail. But note that an RD can be a valid instrument. In the earthquake example, which side of the boundary someone lived on can be an instrument for school attendance because it is plausibly not correlated with anything else that explains the outcomes. Other clever strategies in this category are shift-share and Bartik instruments. These also have research exploring the assumptions they rely on.
4. Difference-in-differences. This method relaxes the assumption of selection on observables. It moves to a before-after setting, and compares the average outcome change of those in the treatment group to the average outcome change of those in the control group. In doing so, the assumption that it makes is that of parallel trends: that the average change of the treatment group would’ve been the same as that of the control group had they not received the treatment. This method is incredibly popular because it’s more robust than selection on observables and settings where it can be credibly applied are more ubiquitous than for regression discontinuity or instrumental variables. A famous example is the minimum wage study of Card and Krueger who compared fast food restaurant workers in the Philadelphia area before and after a minimum wage change. A relatively recent variant of this method is that of synthetic controls which constructs an artificial control group and does diff-in-diff, which you may or may not like for its credibility.

Answer 2

I would like to give you a philosophical and a scientific answer:

In theory and in principle, causality cannot be observed. It never has and never will. Let's take a simple example: when you hit the buttons of your keyboard and the letters appear on your screen whilst typing a post on this website, you assume a causal effect. Firstly, because you observe correlation between you hitting the keys and letters appearing your screen. And secondly, because you have a model of causality of what is happening in your mind which you find plausible (which is basically that the keyboard is an input device used to type).

However, neither of the two are causality and you cannot observe causality. It could be that an invisible demon creates the letters on your screen every time you hit the keys. That is the philosophical point of view and answer.

The scientific answer is to observe causality: you need to manipulate your input data, control for everything else and observe the effect. Since you're not a psychologist designing a study but analyzing data that means you need to have data over time.

So for example if your assumption is that living in a populated city increases the risk of suffering from clinical depression: then you will need a sample of people living in a big city who later developed clinical depression. And not just a positive correlation between the variable "does live in a big city" and "suffers from clinical depression". And you will also need to control for other independent variables.

Another way to achieve this would be in a laboratory setting where you can explicitly manipulate variables (and it is much easier to control for other independent variables). This approach however is not so much related to data science.

Answer 3

Briefly...

Option 1:

Randomized Controlled Trial. The 'gold standard'.

Option 2:

1. Draw a causal diagram of your system. A directed acyclic graph of how you and others think the system operates.
2. Decide if one can infer causation from observational study, by the back door criterion, front door criterion, or other conditional independence methods. Collect data on relevant variables. See Judea Pearl.
3. Build statistical model using 1 & 2.
4. Tred with caution as your DAG, statistical model, nor your data are perfect.

For a gentle introduction see Pearl's The Book of Why

Answer 4

Not sure this adds anything, but if you need another thought from philosophy, back in the day, (1960s) we were taught in a philosophy class that Hume’s 3 criteria of causality required: (1) temporal precedence (presumed cause prior in time); (2) an observable empirical correlation; and (3) that all rival hypotheses had been ruled out.

Assuming criteria #3 to be practically impossible, it would follow causation will be forever impossible to demonstrate.

Answer 5

In short, to detect causation directly, we need to control for everything else. For example, you plant two trees using the same soil, the same amount of water, the same time under the light, and so on but with two different fertilizers. If everything is the same and tree A is growing faster, then we may say that the fertilizer for tree A causes faster development.

We can make that kind of conclusion only we are assuming that everything else is the same. This may be difficult to check so that in practice it is an assumption. For example, two trees may have different genes and one gene causes faster development.

Answer 6

You can not find causation with analysis of the same data which shows correlation.

Sammy above gave an example of hypothesis: living in big cities causes mental disorders. The study he proposes have only two features: location and mental disorder status, and it can show only correlation, not causation. There is always a possibility that people with tendency of mental disorders prefer to live in big cities, and not cities cause disorders.

Some additional attributes have to be involved. These may be attributes which explain the dependence. For example, one may consider a level of noise as an independent variable.

As another option, one may include time in the study, to observe the process, how one is causing another. In particularly, one may consider the same people who lived both in cities and countries in different times of their lives, to see where the disorder occurred more often with these people.

Anyway, there has to be additional information, explaining the causation or registering the process of influence.

Answer 7

I'm going to focus on a narrow topic: what if you can't do a two group experiment, either randomized or observational? What if you have only one group? Or what if you are talking about some national policy change where, because the change happened to the entire country, there's no obvious control group? I think you can attribute causation in some limited circumstances here.

In the clinical setting, health services researchers obviously prefer to conduct randomized clinical trials where possible, and the standard is to conduct a before treatment and after treatment measurement in each arm. In a very limited number of clinical settings, we might be able to make some causal inference in single-arm studies, as discussed by Scott Evans:

...single arm trials are best utilized when the natural history of the disease is well understood when placebo effects are minimal or nonexistent, and when a placebo control is not ethically desirable. Such designs may be considered when spontaneous improvement in participants is not expected, placebo effects are not large, and randomization to a placebo may not be ethical. On the other hand, such designs would not be good choices for trials investigating treatments for chronic pain because of the large placebo effect in these trials.

In my interpretation, say you have some very severe disease. Its mortality rate is well known and pretty high. Say that we know that 80% of patients die within one year of contracting disease X. Say we have a case series (i.e. a set of cases alone, without controls) where patients were given drug Y and we observed a mortality rate of 30%. In that scenario, I think many researchers would be willing to cautiously attribute causation. It might not be viable to conduct a randomized trial. If no two-arm observational studies were available, we would probably be willing to make recommendations based off just a case series.

How does this thinking extend to other scenarios, like the national intervention I mentioned? I think that economists have encountered this scenario more. I think that there are a number of studies about the outcomes associated with Medicaid (in the US, this program provides health insurance for the poor, which is an oversimplification but it will do). The thing is, Medicaid is controlled by the states (as opposed to the Federal, or national, government). Some states expanded Medicaid earlier than others. I believe economists have used this disparity to attempt to attribute causation, but I'm less familiar with that set of methods.

In health services research, hospital checklists are a nice parallel, because of the risk of spillover. You would ideally find, say, 60 hospitals, and randomize 30 of them to start using checklists. This is very hard to pull off. You might be a researcher at one hospital. The only thing you might be able to do is a before vs. after comparison. Here, you probably would want to make the pre- and post-intervention periods as long as you possibly could. I am not familiar with the issues of causation in this sort of scenario.