6
$\begingroup$

Some people in my line of work are interested in the coefficients that results from developing predictive regression models(prm).

I’m somewhat reluctant to use these coefficients to explain an effect a predictor may or may not have on the response. What is sometimes useful is the variable importance, although this seems to be of less interest.

I feel like the difference between predictive and causal analysis is not clear cut for some of the people I report to with my models.

My question is:

  • is it legit to use coefficients from a prm for inference?

The prm's I usually use at work are the following:

  1. Elastic Net/LASSO - Gaussian and logit

  2. Boosted GAMLSS

A lot is telling me that the coefficient are too unreliable and shouldn’t be considered, especially for large models. And if there is a lot of interest for causal effects, then there should be a priority to make additional causal models.

PS: my department is conservative and old fashion with a lot of old people(no offense) and many enjoy working with SAS(offense)

$\endgroup$
  • 3
    $\begingroup$ One note of distinction is that in predictive/prognostic settings, the predictors need not be causally related, only correlated. Causation should therefore be viewed in light of this and any underlying theory for your context. As a general rule, the clinicians I work with are usually not interested in the estimated effect size of any one coefficient, and if so, then a prospective study should be designed to collect this information. $\endgroup$ – prince_of_pears Jan 10 '18 at 22:31
  • 1
    $\begingroup$ Slide 17: statweb.stanford.edu/~tibs/sta305files/Rudyregularization.pdf $\endgroup$ – generic_user Jan 10 '18 at 22:44
3
$\begingroup$

I see two somewhat unrelated questions in this question.

  • Is it possible to draw reliable inference about individual coefficients in predictive models, especially if we have a large number of predictors and use some form of variable selection and/or regularization?
  • Can the coefficients in a predictive model be interpreted causally?

My short answer to the first question is yes, that's possible, but it's not straightforward to do correctly, and it's the subject of intense current research.

To the second question I find that the safe answer is no, the coefficients in a predictive model don't generally have a causal interpretation. This point should be made very clear to collaborators/clients, who may not have a strong training in causal models.

Inference

Emmanuel Candes gave the 2017 Wald Lectures at the Joint Statistical Meetings entitled What's happening in Selective Inference?, which offers a great starting point for learning what the challenges are, and what the status is.

A main challenge, especially when the number of predictors is large, is how to compute and report uncertainty correctly, when the model/predictors have been selected by the data.

Candes explains at length his contributions (mostly with Rina Barber) on the knockoff filter, which is a very nice idea for controlling the false discovery rate of selected predictors.

Another question is how to reliably compute confidence intervals for the coefficients. Candes touches upon this in his talk, but see the paper Exact post-selection inference, with application to the lasso by Lee at al. for more details, and see also the paper Valid post-selection inference by Berk et al.

The R package Selective Inference implements these ideas. Another relevant R package to consider is hdi, see also the paper High-Dimensional Inference: Confidence Intervals, p-Values and R-Software hdi by Dezeure et al.

Note that there is a non-trivial discussion in selective inference about what the target parameter actually is! Is it the (theoretical) coefficient in the model with the selected predictors, or is it the coefficient in the model with all predictors included? Read the paper by Berk et al. for some discussion on this difference.

I would typically investigate uncertainty of reported coefficients and selected predictors via simulations/bootstrapping (remembering to include the full variable selection procedure within the bootstrapping), but it may actually require some work to make sure that e.g. bootstrapped confidence intervals are appropriate, see Bootstrapping Lasso Estimators by Chatterjee and Lahiri.

I should say that the challenges discussed above are fundamentally frequentistic of nature. See e.g. Gelman's post Bayesian inference completely solves the multiple comparisons problem for a more Bayesian perspective.

Causality

Regression models have been used in econometrics and epidemiology, to mention some areas, to estimate causal effects from observational data. This, I find, has historically not always been done with a crystal clear discussion of what actually constitutes a causal effect. Causality has been thought justified by appealing to "no unmeasured confounders" and other similar properties of the setup, in an attempt to argue that the regressors included are precisely those that are needed to justify a causal interpretation of estimated coefficients. But often without a clear conceptual or mathematical framework for defining causality and causal effects.

The history of how Simpson's paradox has been treated in the statistical literature illustrates the problems as described by Pearl in his paper Understanding Simpson’s Paradox.

What is crystal clear to me is that causality is a concept beyond a probabilistic model, and this can be formalized using frameworks such as counterfactuals, structural equation models or graphical models (DAGs). These are not unrelated frameworks, but offer slightly different concepts and languages to introduce the fundamental parameters of interest: the causal effects.

In some situations it might be possible to interpret coefficients from a predictive (regression) model as causal effects, but I would say it's unlikely to be the case if the model is optimized for purely predictive performance from a vast number of potential predictors using observational data.

The forthcoming Causal Inference Book by Hernan and Robins is a great place to learn about the causal models. Part II of the book deals specifically with the use of models for causal inference.

Causal effects can sometimes be estimated using predictive models, but it may require some ingenuity. Inverse probability weighting relies on a predictive model of probability weights, as Hernan and Robins describe. The recent paper Causal inference by using invariant prediction: identification and confidence intervals by Peters, Bühlmann and Meinshausen relies on the causal model being invariant under different (unspecified) interventions, whereas non-causal associations are not.

In any case, I would strongly advice against careless interpretations of (regression) coefficients as causal effects. If causal effects are of interest, this should be taken seriously, and appropriate methods should be employed to estimate the effects of interest.

$\endgroup$
3
$\begingroup$

Frank Harrell, in his 'Regression Modeling Strategies' (2015) offers a range of possible modeling strategies (section 4.12, if you are able to obtain a copy), some of which may be considered facetious ('develop a black box model that performs poorly and is difficult to interpret'), but he then goes on to develop a strategy for regression models which provide accurate predictions, and then discusses how this model may be improved to allow accurate effect estimation, commenting 'estimation of effects for these models must involve accurate prediction of overall response values'. Effectively it seems doubtful that you can have an accurate estimate of the value of a predictor's effect size if you don't have a model that accurately predicts your target.

Harrell points out some useful considerations needed to ensure that a model providing good accuracy can provide good estimates of predictor effects. For example, one is strict attention to interaction effects. Another is the role of imputation for missing data, especially if a variable whose effect size is a high priority has many missing values e.g. it might be sensible to impute the missing values if model accuracy is the only goal, but not sensible if estimating that particular effect is the goal.

At the same time, the above implies that it is possible to achieve a level of accuracy with respect to the target without having achieved accurate estimates of each predictor's effect. You also mention causal analysis, and one of the commenters rightly observes that you can achieve accurate estimates of correlations without accurately understanding causal relationships.

Overall, it begins to look like a hierarchy, where a strong predictive model forms the foundation for accurate estimation of predictor effects, and accurate predictor estimates of predictor effects could then be the beginning of an analysis of causation. Then the answer to your main question is 'yes, it is legitimate to use the coefficients from a predictive model for inference, under the condition that the model has been analysed carefully to ensure the legitimacy of the effects estimates'.

$\endgroup$
  • 1
    $\begingroup$ I’m still not completely convinced. One reason is that you have to trade bias for less variance when you optimize for the best possible predictions, hence you are not getting the BUE. Also there are issues such as multicollinearity and reverse causality that may be problematic for a causal analysis but does not affect the predictions. These kinds of issues are likely to occur in large models and not unlikely to occur in small models. The coefficients in a predictive model may thus be very biased and in worst cases misleading. $\endgroup$ – Nesvold Jan 11 '18 at 7:35
  • $\begingroup$ My contention is that a predictive model that achieves a decent amount of accuracy is a necessary but not sufficient foundation for a model that is useful for inference. Similarly, a model that is useful for inference is a necessary but insufficient condition for a causal analysis. Alternatively, I would put it as 'you must use a good predictive model to analyse for inference, but not every good predictive model will be suitable'. Start with your existing prm and analyse for the problems you mention, and alter it/ reject it if necessary. $\endgroup$ – Robert de Graaf Jan 11 '18 at 7:50
  • $\begingroup$ Predictive models can be helpful to test out results from causal analyses, but I’m not sure if I’m following why predictive models are a necessary foundation for inference. If I was concerned with effects then I would probably make a very different model then if I was concerned predictions. Using predictive optimization methods that produce variable importance is however one way to gain some causal insight, although I wouldn’t use it to verify theories in research per se. Could you refer me to a chapter I Harrels' book? It seems like it’s a very discussable topic :) $\endgroup$ – Nesvold Jan 11 '18 at 8:59
  • $\begingroup$ @Nesvold I've always wondered this, but have never got a solid answer: "One reason is that you have to trade bias for less variance when you optimize for the best possible predictions, hence you are not getting the BUE". Why is unbiasedness so important for this type of applications (causal analysis)? Wouldn't a high variance estimate of an effect, causal or not, be just as bad, maybe worse? $\endgroup$ – Matthew Drury Jan 11 '18 at 19:49
  • $\begingroup$ @MatthewDrury Thats a good point, especially if you are more Bayesian inclined. However asymptotic theory supports the fact that your BUE will in fact converge to the population parameter as long as the variation exists. Still, I think the main issue is very much up for discussion. I’m interested in theory that backs up the use of coefficients in predictive models, even though I’m more inclined to dismiss them. $\endgroup$ – Nesvold Jan 11 '18 at 20:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.