Can cross validation be used for causal inference?

Question

In all contexts I am familiar with cross-validation it is solely used with the goal of increasing predictive accuracy. Can the logic of cross validation be extended in estimating the unbiased relationships between variables?

While this paper by Richard Berk demonstrates the use of a hold out sample for parameter selection in the "final" regression model (and demonstrates why step-wise parameter selection is not a good idea), I still don't see how that exactly ensures unbiased estimates of the effect X has on Y any more so than choosing a model based on logic and prior knowledge of the subject.

I ask that people cite examples in which one used a hold-out sample to aid in causal inference, or general essays that may help my understanding. I also don't doubt my conception of cross validation is naive, and so if it is say so. It seems offhand the use of a hold out sample would be amenable to causal inference, but I do not know of any work that does this or how they would do this.

Citation for the Berk Paper:

Statistical Inference After Model Selection by: Richard Berk, Lawrence Brown, Linda Zhao Journal of Quantitative Criminology, Vol. 26, No. 2. (1 June 2010), pp. 217-236.

PDF version here

This question on exploratory data analysis in small sample studies by chl prompted this question.

Answer 1

I think it's useful to review what we know about cross-validation. Statistical results around CV fall into two classes: efficiency and consistency.

Efficiency is what we're usually concerned with when building predictive models. The idea is that we use CV to determine a model with asymtptotic guarantees concerning the loss function. The most famous result here is due to An asymptotic equivalence of choice of model by cross‐validation and Akaike's criterion (Stone 1977) and shows that LOO CV is asymptotically equivalent to AIC. But, Brett provides a good example where you can find a predictive model which doesn't inform you on the causal mechanism.

Consistency is what we're concerned with if our goal is to find the "true" model. The idea is that we use CV to determine a model with asymptotic guarantees that, given that our model space includes the true model, we'll discover it with a large enough sample. The most famous result here is due to Linear Model Selection by Cross-Validation (Shao 1993) concerning linear models, but as he states in his abstract, his "shocking discovery" is opposite of the result for LOO. For linear models, you can achieve consistency using LKO CV as long as $k/n \rightarrow 1$ as $n \rightarrow \infty$. Beyond linear mdoels, it's harder to derive statistical results.

But suppose you can meet the consistency criteria and your CV procedure leads to the true model: $Y = \beta X + e$. What have we learned about the causal mechanism? We simply know that there's a well defined correlation between $Y$ and $X$, which doesn't say much about causal claims. From a traditional perspective, you need to bring in experimental design with the mechanism of control/manipulation to make causal claims. From the perspective of Judea Pearl's framework, you can bake causal assumptions into a structural model and use the probability based calculus of counterfactuals to derive some claims, but you'll need to satisfy certain properties.

Perhaps you could say that CV can help with causal inference by identifying the true model (provided you can satisfy consistency criteria!). But it only gets you so far; CV by itself isn't doing any of the work in either framework of causal inference.

If you're interested further in what we can say with cross-validation, I would recommend Shao 1997 over the widely cited 1993 paper:

• An Asymptotic Theory for Linear Model Selection (Shao, 1997)

You can skim through the major results, but it's interesting to read the discussion that follows. I thought the comments by Rao & Tibshirani, and by Stone, were particularly insightful. But note that while they discuss consistency, no claims are ever made regarding causality.

Answer 2

This is a really interesting question and I don't offer any specific citations. However, in general, I'd say, NO, in and of itself, cross-validation does not offer any insight into causality. In absence of a designed experiment, the issue of causality is always uncertain. As you suggest, cross-validation can and will improve predictive accuracy. This, alone, says nothing about causality.

Absent of a designed experiment, causal inference would require a model that includes all of the relevant predictors--something that we can rarely guarantee in an observational study. Moreover, a simple lag variable, for example (or anything highly correlated with whatever outcome we were trying to predict) would produce a good model and one which could be validated in multiple samples. That does not mean, however, that we can infer causation. Cross-validation assures repeatability in predictions and nothing more. Causality is a matter of design and logic.

EDIT: Here's an example to illustrate. I could build a model with good predictive accuracy that predicts the population of a city based on the amount of money the city spends on trash removal. I could use cross-validation to test the accuracy of that model as well as other methods to improve the accuracy of prediction and get more stable parameters. Now, while this model works great for prediction, the causal logic is wrong--the causal direction is reversed. No matter what the folks in the Public Works Department might argue, increasing their budget for trash removal would not be a good strategy to increase the city's population (the causal interpretation).

The issues of accuracy and repeatability of a model are separate from our ability to make causal inferences about the relationships we observe. Cross-validation helps us with the former and not with the latter. Now, IF we are estimating a "correct" model in terms of specifying a casual relationship (for example, trying to determine what our trash removal budget should be based on our expected population next year), cross-validation can help us to have greater confidence in our estimate of that effect. However, cross-validation does nothing to help us choose the "correct" model with regard to causal relationships. Again, here we need to rely on the design of the study, our subject matter expertise, theory, and logic.

Answer 3

It seems to me that your question more generally addresses different flavour of validation for a predictive model: Cross-validation has somewhat more to do with internal validity, or at least the initial modelling stage, whereas drawing causal links on a wider population is more related to external validity. By that (and as an update following @Brett's nice remark), I mean that we usually build a model on a working sample, assuming an hypothetical conceptual model (i.e. we specify the relationships between predictors and the outcome(s) of interest), and we try to obtain reliable estimates with a minimal classification error rate or a minimal prediction error. Hopefully, the better the model performs, the better it will allow us to predict outcome(s) on unseen data; still, CV doesn't tell anything about the "validity" or adequacy of the hypothesized causal links. We could certainly achieve decent results with a model where some moderation and/or mediation effects are neglected or simply not known in advance.

My point is that whatever the method you use to validate your model (and holdout method is certainly not the best one, but still it is widely used in epidemiological study to alleviate the problems arising from stepwise model building), you work with the same sample (which we assume is representative of a larger population). On the contrary, generalizing the results and the causal links inferred this way to new samples or a plausibly related population is usually done by replication studies. This ensures that we can safely test the predictive ability of our model in a "superpopulation" which features a larger range of individual variations and may exhibit other potential factors of interest.

Your model might provide valid predictions for your working sample, and it includes all potential confounders you may have think of; however, it is possible that it will not perform as well with new data, just because other factors appear in the intervening causal path that were not identified when building the initial model. This may happen if some of the predictors and the causal links inferred therefrom depend on the particular trial centre where patients were recruited, for example.

In genetic epidemiology, many genome-wide association studies fail to replicate just because we are trying to model complex diseases with an oversimplified view on causal relationships between DNA markers and the observed phenotype, while it is very likely that gene-gene (epistasis), gene-diseases (pleiotropy), gene-environment, and population substructure all come into play, but see for example Validating, augmenting and refining genome-wide association signals (Ioannidis et al., Nature Reviews Genetics, 2009 10). So, we can build-up a performant model to account for the observed cross-variations between a set of genetic markers (with very low and sparse effect size) and a multivariate pattern of observed phenotypes (e.g., volume of white/gray matter or localized activities in the brain as observed through fMRI, responses to neuropsychological assessment or personality inventory), still it won't perform as expected on an independent sample.

As for a general reference on this topic, can recommend chapter 17 and Part III of Clinical Prediction Models, from EW Steyerberg (Springer, 2009). I also like the following article from Ioannidis:

Ioannidis, JPA, Why Most Published Research Findings Are False? PLoS Med. 2005 2(8): e124

Answer 4

This is a good question, but the answer is definitely no: cross-validation will not improve causal inference. If you have a mapping between symptoms and diseases, cross-validation will help to insure that your model matches their joint distribution better than if you had simply fit your model to the entire raw data set, but it can't ever tell you anything about the directionality of causation.

Cross-validation is very important and worth studying, but it does nothing more than prevent you from overfitting to noise in your data set. If you'd like to understand it more, I'd suggest Chapter 7 of ESL.

Answer 5

To respond to the follow-up @Andy posted as an answer here...

Although I could not say which estimate is correct and which is false, doesn't the inconsistency in the Assault Conviction and the Gun conviction estimates between the two models cast doubt that either has a true causal effect on sentence length?

I think what you mean is the discrepancy in the parameter estimates gives us reason to believe that neither parameter estimate represents the true causal effect. I agree with that, though we already had plenty of reason to be skeptical that such a model would render the true causal effect.

Here's my take: Over-fitting data is a source of biased parameter estimates, and with no reason to believe that this bias offsets other sources of bias in estimating a particular causal effect, it must then be better, on average, to estimate causal effects without over-fitting the data. Cross-validation prevents over-fitting, thus it should, on average, improve estimates of causal effects.

But if someone is trying to convince me to believe their estimate of a causal effect from observational data, proving that they haven't over-fit their data is a low-priority unless I have strong reason to suspect their modelling strategy is likely to have over-fit.

In the social science applications I work with, I'm much more concerned with substantive issues, measurement issues, and sensitivity checks. By sensitivity checks I mean estimating variations on the model where terms are added or removed, and estimating models with interactions allowing the effect of interest to vary across sub-groups. How much do these changes to the statistical model affect the parameter estimate we want to interpret causally? Are the discrepancies in this parameter estimate across model specifications or sub-groups understandable in terms of the causal story you are trying to tell, or do they hint at an effect driven by, e.g. selection.

In fact, before you run these alternate specifications. Write down how you think your parameter estimate will change. Its great if your parameter estimate of interest doesn't vary much across sub-groups, or specifications - in the context of my work, that is more important than cross-validation. But other substantive issues affecting my interpretation are more important still.

Answer 6

I thank everyone for their answers, but the question has grown to something I did not intend it to, being mainly an essay on the general notion of causal inference with no right answer.

I initially intended the question to probe the audience for examples of the use of cross validation for causal inference. I had assumed such methods existed, as the notion of using a test sample and hold out sample to assess repeatability of effect estimates seemed logical to me. Like John noted, what I was suggesting isn't dissimilar to bootstrapping, and I would say it resembles other methods we use to validate results such as subset specificity tests or non-equivalent dependent variables (bootstrapping relaxes parametric assumptions of models, and the subset tests in a more general manner are used as a check that results are logical in varied situations). None of these methods meets any of the other answers standards of proof for causal inference, but I believe they are still useful for causal inference.

chl's comment is correct in that my assertion for using cross validation is a check on internal validity to aid in causal inference. But I ask we throw away the distinction between internal and external validity for now, as it does nothing to further the debate. chl's example of genome wide studies in epidemiology I would consider a prime example of poor internal validity, making strong inferences inherently dubious. I think the genome association studies are actually an example of what I asked for. Do you think the inferences between genes and disease are improved through the use of cross-validation (as oppossed to just throwing all markers into one model and adjusting p-values accordingly?)

Below I have pasted a copy of a table in the Berk article I cited in my question. While these tables were shown to demonstrate the false logic of using step-wise selection criteria and causal inference on the same model, lets pretend no model selection criteria were used, and the parameters in both the training and hold out sample were determined A priori. This does not strike me as an unrealistic result. Although I could not say which estimate is correct and which is false, doesn't the inconsistency in the Assault Conviction and the Gun conviction estimates between the two models cast doubt that either has a true causal effect on sentence length? Is knowing that variation not useful? If we lose nothing by having a hold out sample to test our model why can't we use cross-validation to improve causal inference (or I am missing what we are losing by using a hold out sample?)

Answer 7

I guess this is an intuitive way to think about the relation between CV and causal inference: (please correct if I am wrong)

I always think about CV as a way to evaluate the performance of a model in predictions. However, in causal inference we are more concerned with something equivalent to Occam's Razor (parsimony), hence CV won't help.

Thanks.