6
$\begingroup$

I'm caught in a bind about the relationship between theoretical models about how the world works and statistical methods for accurately predicting an outcome in fields where little is known. I strongly suspect that this is due to an incomplete understanding about the correct uses and purposes of regularization / variable selection methods such as Elastic Net and the Lasso, so I'm hoping you guys can help me move forward. Let me get concrete:

I have a p>n dataset about a medical phenomenon on which little -- but not zero -- is currently known (for reference, n=250 and p~350, give or take some qualitative free-text variables that won't factor into the analysis I'm talking about here). Say for argument's sake that the phenomenon of interest, y, is a binary outcome - either it doesn't happen (0) or it does (1).

Broadly, what I'm interested in knowing is which characteristics in this dataset predict y? Model interpretability matters a lot for me here, whatever the approach. I'm not just interested in successfully predicting y, but that I want to know which variables contribute to doing so.

To find out, I can think of two broad approaches:

  1. (a) make a theoretical model based on what little prior subject-matter knowledge of y there is, which I then (b) turn into a linear combination of a parsimonious subset of all 350 predictors (say e.g. 10 of them) and regress y on that subset in a more-or-less vanilla manner and finally (c) evaluate the performance of that model in some acceptable way (R^2; statistical significance of predictor betas; accuracy on a testing set of a model fit on a training set; etc.) and write up the result (e.g. "we found that variable A was a statistically significant predictor of y, but that variables B and C were not" -- this would be the standard/traditional social science approach, or maybe "after training the model on a randomly selection proportion of the data, its performance on a testing set of "new" data was ____" -- this would be the more modern approach).

  2. (a) forgo any prior model and (b) use a sort of 'kitchen sink plus regularization' approach in which y is regressed onto all 350 predictors and a regularization / variable selection method like Elastic Net is used (in combination with k-fold CV to select optimal tuning parameters) to "find" a model - its own subset of predictors - which maximizes prediction accuracy.

Here's my dilemma, then: what if I do both of these and find that (2) is not only better at predicting y than my own hypothesized (1), but that the predictors selected by (2) are different (partially or entirely) from the ones I selected a priori? What does that mean for the phenomenon as it exists out there in nature -- is Elastic Net saying something about "nature" at all? Would I have to throw out my own model, or figure out a way to account for any "new" variables that I hadn't hypothesized would matter but that the regularization method is telling do matter?

Would it make any difference if, in Approach (2), instead of throwing all 350 variables into the model, I threw the subset of predictors I used in Approach (1) -- or would that be changing the whole question?

Thanks!

This question is closely related to another asked on CV [1, 2], but it isn't quite the same, I don't think. If this has been addressed in the past, I apologize for having overlooked it!

1 2

$\endgroup$
3
  • 1
    $\begingroup$ Possibly relevant for #2: Sane stepwise regression? $\endgroup$ Dec 19, 2021 at 16:11
  • 1
    $\begingroup$ How many positive & negative cases are there out of the 250? Can you get a continuous variable instead of the binary one? Is there really no scientific knowledge that can be brought to bear? It may be that nothing can be done here. $\endgroup$ Dec 19, 2021 at 16:27
  • 1
    $\begingroup$ To me, this sounds like a conflating of prediction and causal inference. If your goal is to describe nature in some capacity, as you say, then why is your question "I want to know what variables simply predict Y?" That's just pure association. Wouldn't a more accurate question be, "what variables X cause Y?" or "what variables X have the largest effect on Y?". Now you have to worry about confounding, colliders, causal identification, etc. as well as making sure the model you are using will produce a consistent and preferably efficient estimator of the causal effect. $\endgroup$
    – aranglol
    Dec 19, 2021 at 18:09

2 Answers 2

2
$\begingroup$

Doubly robust methods (Urminsky et al. "Using Double-Lasso Regression for Principled Variable Selection") have become very popular recently since they allow (see page 18, Concluding Remarks) "identifying which covariates to include and not include in analyses" (even if the number of variables is larger than the sample size as in your case).

This empirical approach alone cannot solve your problem (and I think none will do so entirely) since you will need some theory (in my view, and in the view of the well-known authors cited above [p. 18]):

the analytic method presented here cannot determine either the role that selected variables should play, or how their effects on the relationship of interest should be interpreted. A confound, a manipulation check and a mediator may all have similar statistical relationships in the data (MacKinnon, Krull, & Lockwood, 2000; Zhao, Lynch, & Chen, 2010), and these distinctions should typically be made on theoretical grounds.

That means regarding your question elastic net, or any empirical approach alone, will not necessarily say anything about the "nature".

But the double robust approach might be still what you are looking for [p. 18]:

However, either including all covariates or ignoring covariates entirely, either because of the conceptual difficulty of identifying the theoretical role of the variable or because of the potential for covariates to be used improperly (i.e., in p-hacking), is no solution. Failing to control for valid covariates can yield biased parameter estimates in correlational analyses or in imperfectly randomized experiments and contributes to underpowered analyses even in effectively randomized experiments. As demonstrated in the analyses, double lasso variable selection can be useful as a principled method to identify covariates in analyses of correlations, moderation, mediation and experimental interventions, as well as to test for the effectiveness of randomization. While variable selection methods are no substitute for thinking about what the variables mean, the approach presented here can provide an empirical basis for determining which variables to think hard about.

There are also R-packages available.

Why a double robust method, and not let's say Lasso alone [p. 5]?

The goal is to identify covariates for inclusion in two steps, finding those that predict the dependent variable and those that predict the independent variable. The second step is important, because exclusion of a covariate that is a modest predictor of the dependent variable but a strong predictor of the independent variable can create a substantial omitted variable bias.

$\endgroup$
3
  • 2
    $\begingroup$ Are you sure you are talking in the right context here? Are doubly-robust method not more about causal inference and propensity scores? I.e. you either need to get the propensity score model right or the covariate adjusted analysis model that uses the propensity score, but not both? How is that relevant to the original question? $\endgroup$
    – Björn
    Dec 23, 2021 at 10:24
  • 1
    $\begingroup$ The original question is indeed about a causal relationship ("I'm not just interested in successfully predicting y, but that I want to know which variables contribute to doing so.") and there are different doubly-robust methods available. The one I linked does not use propensity score matching. Therefore I see my answer as relevant. $\endgroup$ Dec 23, 2021 at 10:46
  • 1
    $\begingroup$ What variables contribute to a prediction $\neq$ causality. $\endgroup$
    – Björn
    Dec 23, 2021 at 15:06
1
$\begingroup$

Prediction models are about making good predictions. That's what you are optimizing (in terms of the metric you optimized your elastic net parameters for), when you go for your second option. Whatever hyperparameter settings help the model predict well as assessed by k-fold CV gets used and then you get some resulting coefficients that are non-zero. You should really not overinterpret those, because post-model-selection inference is difficult. There's quite a bit of literature about post-selection inference that try to find ways of doing this that are in some sense "valid", but it's tricky. Certainly, with the numbers you describe you would expect to miss out on some in truth quite relevant predictors by chance. There's also some serious risk that some spurious predictors end up in your model, but that's where methods for post-selection inference would come in to limit that to some degree.

However, don't expect too much. You have a tiny dataset and realistically only so much can be done (see the 2nd quote here: https://en.wikiquote.org/wiki/John_Tukey).

The first approach is less problematic for interpreting the coefficients, because you at least do not have the model selection messing everything up in terms of interpretation. However, you should still be careful to not overinterpret statistical significance (firstly, important predictors might not be by chance and less important ones can be, secondly you of course have a multiple comparison problem) or the coefficients (due to the small sample size even changing sign - aka type S error - or simply completely getting the magnitude wrong - aka type M error - are very real issues).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.