# Simulating data - correlation vs causation

I'm just dipping my toe into correlation vs causation, so forgive me if I butcher some of the concepts here.

To get a better understanding of these concepts from data, I would like to simulate data under a correlational assumption and a causational assumption. I'm not sure where to start, but here's where I'm headed.

For correlation, this is fairly easy.

• Generate a sequence of normal random variables to serve as a basis.
• Duplicate that sequence but add random normal noise to the variable - essentially produce a new sequence using aX + b, with a being a single random value, and b being a new sequence of small random normal values to introduce noise. The magnitude of a and b will determine the final correlation.

If I do this multiple times, I'll have a set of variables that are all correlated with the original sequence and will likely have correlation structure among each other. If I remove the original sequence, the remaining sequences will have some degree of pairwise correlation - the covariance matrix will be non-diagonal - but there will be no causation between them.

For causation, my thoughts were:

• Choose a distribution (Normal, Gamma, Beta, ...)
• For the parameters of that distribution, draw a random variable from another distribution - standard normal is probably fine.
• Generate a sequence of random variables from the first distribution using the determined parameters determined from the second.

In my mind, this creates a causal relationship, but I'm sure I'm wrong. Regardless, the result is a data matrix that has some correlation - different variables from the same first distribution with parameters drawn from the second distribution may have a correlation but they are not related to each other causally.

I appreciate any comments/suggestions regarding how to approach this.

• My understanding about this issue is fundamental but from what I read during my learning of directed acyclic graph and causal influence, when simulating causally related variables, we only also rely on correlation. In fact, data alone are insufficient, to support causality one will have to take study design and plausible mechanism into account as well. – Penguin_Knight Nov 8 '17 at 14:54
• To echo Penguin_Knight, the data itself are not causal or correlational on the basis of correlation alone. The real context in which the data were generated matters. – RickyB Nov 8 '17 at 15:09

All this is easier with a theory of causality. For example, let's use here the Structural Causal Models (which includes the Potential Outcomes) approach.

A Structural Causal Model (SCM) is triplet $M = \langle V, U, F\rangle$ where $U$ is a set of exogeneous variables, $V$ a set of endogenous variables and $F$ is a set of structural equations that determines the values of each endogenous variable. The structural equations are in the sense of assignments not equalities. For example, consider the simple structural equation $Y = X^2$. This is meant to be read $Y\leftarrow X^2$, in the sense that if I experimentally set $X=2$ then this causally determines the value of $Y = 4$ but experimentally setting $Y = 4$ does nothing to $X$. The asymmetry is important/fundamental in causality: rain causes the floor to be wet, but making the floor wet does not cause rain.

So our causal model can be thought as functional relationships among variables and we are considering these relationships as autonomous. You can think of it as an idealized representation of the real world, where the variables $V$, the endogenous variables, are what we choose to model, and the variables $U$ are the aspects we chose to ignore. Since we chose not to model the $U$, what we usually do is to represent our ignorance about $U$ with a probability distribution $P(U)$ over the domain of $U$, giving us a probabilistic SCM which is pair $\langle M, P(U) \rangle$. Notice this means that causal relationships are ultimately functional relationships, therefore causal relationships may or may not translate to specific probabilistic dependencies. Finally, every causal model can be associated with a directed (acyclic) graph $G(M)$.

Hence, one way to simulate from a probabilistic causal model is by specifying: (i) the endogenous variables $V$ you are going to model; (ii) the exogenous variables $U$ which are usually the "disturbances", along with their joint probability distribution; and, (iii) the (causal) structural relationships among the variables. It might be easier to start this process qualitatively by first drawing the causal DAG with the main features that you want to illustrate and then add the details of the simulation (functional forms) later.

To see how this can be easily done in practice, let's simulate a simple causal model that illustrates simpson's paradox (for more see Pearl). Suppose our model $M$ is given by the following causal DAG, where the variables in parenthesis are "unobserved" and each variable has an associated exogenous disturbance $U$ which is omitted for convenience:

More specifically we will assume the following structural equations $F$:

\begin{aligned} W_1 &= U_{W_1}\\ W_2 &= U_{W_2}\\ Z &= W_1 + W_2 + U_{Z}\\ X &= W_1 + U_{x}\\ Y &= X+ 10W_2+ U_{y}\\ \end{aligned}

Finally, assume all disturbances in $U$ are independent standard normal random variables. Now it's easy to simulate from our causal model. In R for instance:

rm(list = ls())
set.seed(1)
n  <- 1e5
w1 <-  rnorm(n)
w2 <-  rnorm(n)
z <-  w1 + w2 + rnorm(n)
x  <- w1 + rnorm(n)
y  <- x + 10*w2 + rnorm(n)


This example is interesting because if you run the regression $Y \sim X$ you get $1$:

lm(y ~ x)

Call:
lm(formula = y ~ x)

Coefficients:
(Intercept)            x
0.01036      1.00081


But if you further "control" for $Z$, which is a pre-treatment variable correlated with both $X$ and $Y$ --- and some people still erroneously would say it's a confounder --- you will actually see a sign reversal of the estimate and get $-1$:

lm(y ~ x + z)

Call:
lm(formula = y ~ x + z)

Coefficients:
(Intercept)            x            z
0.00845     -1.01127      4.00041


In this example, since we simulated the data, we know the true causal effect is $1$ which is captured by the first regression. But you can only know that if you know the true causal structure. There's nothing in the data itself that tells you which one is the correct answer. Hence if you simulate this and give to a researcher only the variables $x$, $y$ and $z$ he can't tell the right answer just from looking at the correlations. If you want further play/simulate causal models with multi-stage Simpson's paradox reversals, you can check it here.

Simulating correlations/dependencies

To simulate correlations/dependencies you can take a similar approach. You can simply create a causal model that gives you the correlations/dependencies you want (adding latent variables if needed), simulate as above and the resulting data will have the desired correlations/dependencies. To make it easier, you can start by drawing the causal DAG (bayesian network) and read from the graph if the desired conditional dependencies/independencies are implied by your model. After that you might think of specific functional forms to get other quantitative aspects that you want. Notice that several models with different causal interpretations can give you the same correlations.

• Thanks for the detailed post! I'll need to read it thoroughly to understand... – KirkD_CO Nov 9 '17 at 3:49
• +1: Terrific answer, as always. One clarification question. Is it the case that when we do not condition on the collider $y_i=\beta_0+\beta_1x_i+\epsilon_i$, the assumption $E(\epsilon_i|x_i)=0$ holds and we get the correct causal effect, but when we erroneously condition on the collider $y_i=\beta_0+\beta_1x_i+\beta_2z_i+\epsilon_i$, we get the incorrect causal effect of x on y and therefore we can conclude that it must be that $E(\epsilon_i|x_i)\not=0$? How can I see that $E(\epsilon_i|x_i)\not=0$ in the second regression? – ColorStatistics Mar 1 at 17:44
• And more broadly, I am asking myself if every time regression does not correctly estimate the causal effect, it must be that $E(\epsilon_i|x_i)\not=0$? – ColorStatistics Mar 1 at 18:00
• @ColorStatistics that's correct, in this particular model, not conditioning in anything gives the correct answer. Regarding knowing that the second regression is wrong, unfortunatelly this is impossible with the data alone. There's nothing on the data that tells us the second regression is wrong and the first regression is right. – Carlos Cinelli Mar 5 at 20:31

First, causality cannot be observed in the output. Only correlation.

You can imagine two functions that output exactly the same $(x_1,x_2)$ pairs (given the same random seed). Yet you could interpret one function as creating causally related pairs, the other not. It depends on how the code is written (or how you interpret it), not what it actually computes as a final result.

Causality as defined with DAGs requires the possibility of intervention on one of the two variables at least as a thought experiment. Imagine it as a debugger: you interrupt your program just after the first variable was computed, reset it to a certain value, and then restore execution. Will this impact the second variable?

In the way you explain the process (second case) call:`

• $A$: choice for the distribution
• $B$: choice for its parameters
• $(X_1,X_2)$: final output

The DAG is :

There is no causal relationship between $X_1$ and $X_2$.

• exactly. X1 and X2 are likely correlated, but not causally related. However, A and B are causally related to X1 and X2, correct? Your graphical model looks like a Bayesian Net idea, which is consistent with my question. – KirkD_CO Nov 8 '17 at 21:07
• Yes. The graph here is a "direct acyclic graph." (DAG). You can read more about it here: stats.stackexchange.com/questions/45999/… or in Carlos answer. SCM and DAG are equivalent ways to define causality (as far as I know). Actually those theories are rather modern and still presented in several different ways but importantly they show how causality can't be defined by probabilities alone but can be studied together with probabilities. The graph + the joint distribution is the full causal model. – Benoit Sanchez Nov 9 '17 at 11:05
• I'm disappointed that I cannot vote both of these as answers. Both have been truly helpful. – KirkD_CO Nov 11 '17 at 22:18