Correlation, regression and causal modeling

Question

This is probably a blindingly obvious answer for any seasoned statistician, but I am still confused as to how correlation differs from regression, technically.

I understand that one is a measure of association and one a measure of causation, but how can you actually measure causation mathematically, without actually conducting a real life experiment.

Answer 1

Correlation vs Regression

I am still confused as to how correlation differs to regression, technically. I understand that one is a measure of association and one a measure of causation

This is incorrect, the difference between correlation and regression is not causal. Both measures are associational measures. I will elaborate on the difference between associational and causal quantities below, but let's quickly answer this part of your question.

Mathematically, the correlation of $X$ an $Y$ is a symmetric quantity (that is, $cor(Y, X) = cor(X, Y)$) and it's given by:

$$ cor(Y, X) = \frac{cov(Y, X)}{sd(Y)sd(X)}$$

Let $R_{yx}$ denote the regression coefficient of regressing $Y$ on $X$. This is usually not symmetric and it's given by:

$$ R_{yx} = \frac{cov(Y,X)}{var(X)} $$

Notice that if $var(X) = var(Y) = 1$ then the correlation coefficient and the regression coefficient will be the same.

These are pure associational quantities and you can see more about them here. Now let's move on to the causal inference part.

Association vs Causation

Let's start by stating what's the difference between association and causation. Consider two random variables, $X$ and $Y$ and now consider these two different questions:

1. What's the expected value of $Y$ if I see $X = x$? Let's denote this by $\mathbb{E}[Y|X=x]$.
2. What's the expected value of $Y$ if I set $X = x$? Let's denote this by $\mathbb{E}[Y|do(X = x)]$

The first question is what regression can always give you. It's an associational question. The second question is an interventional question --- what would happen if you could set the value of $X$ to whatever you please? Usually, this is not the same as regression.

Let's see an example.

Consider the following structural equations:

$$ U = \epsilon_{u}\\ X = \delta U + \epsilon_{x}\\ Y = \beta X + \gamma U + \epsilon_y $$

Where all terms denoted by $\epsilon$ are mean zero and mutually independent gaussians. To simplify computation, also assume $U$, $X$ and $Y$ have been standardized (mean zero and unit variance). Suppose $U$ is unobserved.

What is the expected value of $Y$ if we observe $X = x$? This is a traditional statistics questions and it's simply:

$$\mathbb{E}[Y|X=x] = \left(\beta + \gamma\delta\right)x$$

And you can estimate $b =\left(\beta + \gamma\delta\right)$ with a linear regression of $Y$ on $X$.

But what is the expected value of $Y$ if we set $X$ to $x$? Setting $X$ to $x$ means erasing the structural equation for $X$ and substituting for $X = x$. Hence:

$$\mathbb{E}[Y|do(X=x)] = \beta x$$

And $\beta$ will be different from the regression coefficient $b$ unless either $\gamma$ or $\delta$ are equal to zero.

When can you estimate causal quantities with regression?

Now let's answer your question:

how can you actually measure causation mathematically, without actually conducting a real life experiment.

In our example, we can't estimate the causal effect of $X$ on $Y$ because there is an open confounding path ($X \leftarrow U \rightarrow Y$), shown in red in the causal diagram below. We usually call this a backdoor path.

$\hskip2.5in$

However,notice that if we could observe $U$ we could recover the causal parameter $\beta$ with a regression conditioning on $X$ and $U$.

More generally, the problem of recovering causal effects with adjustment from regression has been mathematically solved. You can recover the structural coefficient from observational data if you can find a set of variables that: (i) blocks all backdoor paths from $X$ to $Y$; and, (ii) do not open other confounding paths (if you want total effects, you also do not want to control for mediators).

But how can you know which variables satisfy (i) and (ii)? You can only know that with causal assumptions. For example, in the model below you don't want to adjust for $Z$!

$\hskip1in$

Take a look here for another discussion about confounders.

That is, we need to know the causal graph (or equivalently, a set of structural equations) in order to tell which set of variables you can use to identify the effect via regression (if that set exists). You need causal assumptions to draw causal conclusions from observational data.

So to sum up

• Correlation coefficients and regression coefficients are different associational quantities, their difference has nothing to do with causality;
• Also, regression is usually not equal to causal quantities. Regression asks: what if I observe X? Causal inference asks: what if I manipulate X?
• But, under some circumstances, you can use regression to identify causal quantities with observational data. In order to do that you need some causal assumptions, for example, to identify when a group of variables satisfy the back-door (or single-door) criterion.

It's also worth noticing that adjustment via regression is not the only way to identify causal effects. For example, two other widely known methods are the front-door criterion and instrumental variables.

If you want to learn more about this, you should check the references here.