4
$\begingroup$

Suppose we are given a dataset but not the capability of performing some AB testing. We do some regression using X as predictor and Y as response and get a model. Can we actually say something about the causal relationship between X and Y? Or is it simply impossible to say anything about the causal relationship at all?

For example, suppose the data we have is simply the fathers' height and the sons' height, and also suppose mother's height has no influence on son's height. We can get a good linear relationship using sons's height as X and fathers' height as Y. However, we cannot say that lower sons' height causes fathers' height to be lower.

In other words, I feel the causal inference has to eventually resort to some physical/mechanical mechanism instead of just by looking at the data. Am I missing something here?

$\endgroup$
  • 1
    $\begingroup$ Well yea, you just can't make causal arguments like "the user's action caused the ad variant". I'm not too familiar with the causal modelling literature, but in my limited exposure, most causal models are based in some sort of mechanistic or well informed arguments. $\endgroup$ – Demetri Pananos Dec 24 '18 at 4:43
  • 3
    $\begingroup$ Clearly not on such a correlation/ regression model alone. E.g. I would expect that in a model to predict whether a ship sinks on a given day lifeboat usage on the same day would be a good predictor, but I suspect that lifeboat usage does not cause the ship to sink. $\endgroup$ – Björn Dec 24 '18 at 7:23
7
$\begingroup$

Suppose we are given a dataset but not the capability of performing some AB testing. We do some regression using X as predictor and Y as response and get a model. Can we actually say something about the causal relationship between X and Y?

No you can't, even when all variables are observed, see here for instance. If you are given only distributional information about the data (ie., you know the joint distribution of the observed variables), but no information about how the data was generated (a causal model), causal inference is impossible. In short, you need causal assumptions to get causal conclusions. You can get started on learning causal inference with the references here.

It is easy to understand why that is the case by constructing an example where different causal models entail the same observed joint probability distribution. Consider that you have observed the joint probability distribution $P(x, y)$ of two random variables. Here, imagine you have no sampling uncertainty---so you have perfect knowledge of $P(x, y)$, which entails perfect knowledge of the regression function and so on. To simplify things, consider that, in your data, $P(x,y)$ was found to be jointly normal with mean $0$, variance 1 and covariance $\sigma_{xy}$ (this is without loss of generality, you can always standardize the data). What can you say about the causal effect of $x$ on $y$ or vice-versa?

With only this information, nothing. The reason here is that there are several causal models that would create the same observed distribution, yet have different interventional (and counterfactual) distributions. Here I will show three of such models. Notice all of them gives you the same observed $\sigma_{xy}$, but their causal conclusions are different: in the first model $X$ causes $Y$, in the second model $Y$ causes $X$, and, in the third model, neither causes each other --- $X$ and $Y$ are both common causes of the unobserved variable $Z$.

Model 1

$$ X = u_{x}\\ Y= \sigma_{yx}x + u_{y} $$

Where $U_{x} \sim \mathcal{N}(0, 1)$ and $U_{y} = \mathcal{N}(0, 1 - \sigma_{xy}^2)$.

Model 2

$$ Y = u_{y}\\ X = \sigma_{yx}y + u_{x} $$

Where $U_{x} \sim \mathcal{N}(0, 1 - \sigma_{xy}^2)$ and $U_{y} = \mathcal{N}(0, 1)$.

Model 3

$$ Z = U_{z}\\ X = \alpha Z + U_{x}\\ Y = \beta Z + U_{y} $$

Where $\alpha\beta= \sigma_{xy}$, $U_{z} = \mathcal{N}(0, 1)$, $U_{x} = \mathcal{N(0, 1- \alpha^2)}$ and $U_{y} = \mathcal{N(0, 1- \beta^2)}$.

$\endgroup$
  • $\begingroup$ +1 cause these are useful examples but I do think you this approach is a bit unnecessarily strict and thus somewhat pessimistic. Causal discovery work (e.g. Peters et al. 2014) does exist; I try to be a bit more optimistic in my answer! :) (I see my post as being complementary to yours). $\endgroup$ – usεr11852 Dec 26 '18 at 2:51
  • $\begingroup$ @usεr11852 causal discovery algorithms rely on causal assumptions, so it boils down to the same motto - no causes in, no causes out. $\endgroup$ – Carlos Cinelli Dec 26 '18 at 5:21
  • $\begingroup$ Erm... In principal yes, I am not dismissing your points. In practice though, if we can reasonably estimate ATE/ITEs doesn't the question of causal discovery becomes redundant? $\endgroup$ – usεr11852 Dec 26 '18 at 13:43
  • 1
    $\begingroup$ @usεr11852 to estimate the ATE you need a causal model — the most common model is to assume ignorability of treatment assignment conditional on observables, see here stats.stackexchange.com/questions/381467/… and here stats.stackexchange.com/questions/182222/… $\endgroup$ – Carlos Cinelli Dec 26 '18 at 20:29
  • $\begingroup$ I am not sure about the need for a causal model. This seems to be like the conversation pre-ML where people tried to define generative models as the only valid way to model because they are interpretable. And then ML (RF/GBM/NN) came over and obliterated these questions by dominating most predictive analytics tasks. And the unconfoundedness that you mention is an assumption, not a model on its own. It does not dictate any causal relation or a path on an SCM. $\endgroup$ – usεr11852 Dec 27 '18 at 1:05
1
$\begingroup$

From data alone, it's impossible. There could always be some factor outside the model that could influence both $X$ and $Y$ (or one of them). It's impossible to control for literally everything.

The closest we have is a randomized control experiment, but even that has problems with external validity (e.g. we assume that what happened and the conditions during the time of the experiment will persist into the indefinite future).

There is 'Granger causality' (which is not true causality), which basically says if the parameters on the lagged $X$ variables in a regression of $Y(t)$ on $X(t-1), ..., X(t-m), Y(t-1), ..., Y(t-m)$ are jointly significant, then $X$ 'Granger causes' $Y$. See Granger (1969).

$\endgroup$
1
$\begingroup$

Potentially. Your intuition about the necessity to "resort to some physical/mechanical mechanism" is correct but that does not mean that the explicit definition of such mechanism is required. We can relax this problem.

There is a lot of work on causal inference from observational data where we do not explicitly formulate the causal model in form of a clear parametric equation. There are "ML-flavoured" approaches like: "Learning Representations for Counterfactual Inference" by Johansson et al., "Causal inference by using invariant prediction" by Peters et al., "Causal Forests" by Athey and various collaborators that make significant inroads. Let's be clear: these approaches require substantial amounts of data and are far from prime-time ready. Nevertheless they offer evidence that while using observational data to answer causal questions is risky, obtaining answers is not impossible.

Final note: we have only recently have started coming up with "causal datasets" - datasets, where we have carefully annotated causal effects. The grand revolution in Computer Vision came through the abundance of available label training data. Causal inference work so far is not enjoying such a data-rich environment to work. Initiatives like the Causality workbench, the Causal Inference challenges, the Tubingen datasets observational samples give us test-beds that were simply unavailable only 10 years ago.

$\endgroup$
  • 2
    $\begingroup$ Some of these methods rely on very strong assumptions. For example, Causal Forests assumes unconfoundedness of treatment given observables, which is unlikely to hold in most non-experimental settings. $\endgroup$ – Dimitriy V. Masterov Dec 26 '18 at 3:22
  • $\begingroup$ @DimitriyV.Masterov: I fully agree. That being said, I haven't seen a method not using very strong assumptions. What would be a method that does not "assume unconfoundedness of treatment given observables" and is reasonably applicable in a non-experimental setting? $\endgroup$ – usεr11852 Dec 26 '18 at 3:34
  • $\begingroup$ (And to state the obvious: Certain theories might make unrealistic assumptions assumptions but ultimately what matters are the predictions/inference made by the theory. All assumptions are not created equal; for example, the normality of residuals assumption for Lin. Regression is always there but realistically mild violations of it never invalidated an otherwise consistent analysis. The same cannot be said for the homoskedasticity or the serial independence of the residuals.) $\endgroup$ – usεr11852 Dec 27 '18 at 1:07
  • $\begingroup$ Prediction of counterfactuals is difficult since you don't observe the other outcome, so most of these strong assumptions are unverifiable. $\endgroup$ – Dimitriy V. Masterov Dec 27 '18 at 1:45
  • 1
    $\begingroup$ There are regression discontinuity, instrumental variables, panel methods like difference in differences, synthetic cohort methods that don't make this assumption, though they make other ones. I merely wanted to say that you cannot use CFs in the OP's problem without making some other assumptions. $\endgroup$ – Dimitriy V. Masterov Dec 27 '18 at 2:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.