How to identify natural experiments?

Question

The motivation of this question is learn what to look for in deciding whether observational data can be used for causal inference (in the sense of Pearl).

Wikipedia describes a natural experiment:

A natural experiment is an empirical study in which individuals (or clusters of individuals) are exposed to the experimental and control conditions that are determined by nature or by other factors outside the control of the investigators. The process governing the exposures arguably resembles random assignment. Thus, natural experiments are observational studies and are not controlled in the traditional sense of a randomized experiment (an intervention study). Natural experiments are most useful when there has been a clearly defined exposure involving a well defined subpopulation (and the absence of exposure in a similar subpopulation) such that changes in outcomes may be plausibly attributed to the exposure. In this sense, the difference between a natural experiment and a non-experimental observational study is that the former includes a comparison of conditions that pave the way for causal inference, but the latter does not.

I find this too vague to translate into actions or decisions.

This post gives a similar description:

The difference is that a natural experiment is an observational study that, in which out so happens that there is something that is effectively a randomization.

So the key ingredient seems to be about whether exposure to a condition was randomized. The same post gives an example:

A nice clean example is studying the effects of suddenly having lots of money by comparing lottery winners against a representative random sample of people that bought tickets for the same drawing. Effectively, someone else did the randomization for you and it very clearly had nothing to do with some characteristic of the people whether they won or not.

Which indeed, this example has randomization explicitly by construction. Looking through the examples on the wikipedia page, I find that some of them are similarly obvious (e.g. lotteries by design) while others I am less sure of.

A phrase from the wiki that doesn't seem to entirely match the above descriptions is:

While game shows might seem to be artificial contexts, they can be considered natural experiments due to the fact that the context arises without interference of the scientist.

I wouldn't think that the scientists not interfering would be enough to conclude that an observational study is a natural experiment.

---

What should we look for in deciding on whether exposure in an observational study was "randomly assigned"?

Answer 1

Randomisation is used because it breaks back-door paths into the effect of interest. The important thing is not randomisation itself - it's the absence of back-door paths. That is, there are no confounders, a.k.a. common causes, that influence both the treatment and the effect. (And no selection bias.)

Causal processes are almost never fully observed, so it's hard to be certain no back-door paths exist unless you randomise. However, in natural experiments we can argue that a back-door path is:

1. Unlikely to operate,
2. Controlled for (i.e. there is an observed covariate we can condition on to block the path), or
3. Too weak to account for the observed effect.

One example given on Wikipedia is John Snow's investigation of cholera mortality among customers of the Southwark and Vauxhall Waterworks Company, vs the Lambeth Waterworks Company. Water service was not assigned randomly, but renters were usually constrained to use whichever company had run pipes to their home, and many had no idea which company they were actually paying. The two distributors serviced slightly different areas, so Snow looked at mortality in the areas where they overlapped (controlling for the possible geographic confounder). There was a striking difference in cholera rates between the customers of each company, too large to explain by other differences between the two groups.

If we're not completely certain of a single natural experiment, we can use convergent evidence to strengthen the case. By itself, the Southwark & Vauxhall vs Lambeth experiment might not have sufficed to show that cholera was waterborne. But in combination with other natural experiments - e.g. mortality among people who drank from the Broad St Pump vs other pumps - Snow had a strong argument.