3
$\begingroup$

I'm caught in a bind about the relationship between theoretical models about how the world works and statistical methods for accurately predicting an outcome in fields where little is known. I strongly suspect that this is due to an incomplete understanding about the correct uses and purposes of regularization / variable selection methods such as Elastic Net and the Lasso, so I'm hoping you guys can help me move forward. Let me get concrete:

I have a p>n dataset about a medical phenomenon on which little -- but not zero -- is currently known (for reference, n=250 and p~350, give or take some qualitative free-text variables that won't factor into the analysis I'm talking about here). Say for argument's sake that the phenomenon of interest, y, is a binary outcome - either it doesn't happen (0) or it does (1).

Broadly, what I'm interested in knowing is which characteristics in this dataset predict y? Model interpretability matters a lot for me here, whatever the approach. I'm not just interested in successfully predicting y, but that I want to know which variables contribute to doing so.

To find out, I can think of two broad approaches:

  1. (a) make a theroretical model based on what little prior subject-matter knowledge of y there is, which I then (b) turn into a linear combination of a parsimonious subset of all 350 predictors (say e.g. 10 of them) and regress y on that subset in a more-or-less vanilla manner and finally (c) evaluate the performance of that model in some acceptable way (R^2; stasitstical significance of predictor betas; accuracy on a testing set of a model fit on a training set; etc.) and write up the result (e.g. "we found that variable A was a statistically significant predictor of y, but that variables B and C were not" -- this would be the standard/traditional social science approach, or maybe "after training the model on a randomly selection proportion of the data, its performance on a testing set of "new" data was ____" -- this would be the more modern approach).

  2. (a) forgo any prior model and (b) use a sort of 'kitchen sink plus regularization' approach in which y is regressed onto all 350 predictors and a regularization / variable selection method like Elastic Net is used (in combination with k-fold CV to select optimal tuning parameters) to "find" a model - its own subset of predictors - which maximizes prediction accuracy.

Here's my dilemma, then: what if I do both of these and find that (2) is not only better at predicting y than my own hypothesized (1), but that the predictors selected by (2) are different (partially or entirely) from the ones I selected a priori? What does that mean for the phenomenon as it exists out there in nature -- is Elastic Net saying something about "nature" at all? Would I have to throw out my own model, or figure out a way to account for any "new" variables that I hadn't hypothesized would matter but that the regularization method is telling do matter?

Would it make any difference if, in Approach (2), instead of throwing all 350 variables into the model, I threw the subset of predictors I used in Approach (1) -- or would that be changing the whole question?

Thanks!

This question is closely related to another asked on CV [1, 2], but it isn't quite the same, I don't think. If this has been addressed in the past, I apologize for having overlooked it!

1 2

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.