I'm caught in a bind about the relationship between theoretical models about how the world works and statistical methods for accurately predicting an outcome in fields where little is known. I strongly suspect that this is due to an incomplete understanding about the correct uses and purposes of regularization / variable selection methods such as Elastic Net and the Lasso, so I'm hoping you guys can help me move forward. Let me get concrete:
I have a p>n dataset about a medical phenomenon on which little -- but not zero -- is currently known (for reference, n=250 and p~350, give or take some qualitative free-text variables that won't factor into the analysis I'm talking about here). Say for argument's sake that the phenomenon of interest, y, is a binary outcome - either it doesn't happen (0) or it does (1).
Broadly, what I'm interested in knowing is which characteristics in this dataset predict y? Model interpretability matters a lot for me here, whatever the approach. I'm not just interested in successfully predicting y, but that I want to know which variables contribute to doing so.
To find out, I can think of two broad approaches:
(a) make a theroretical model based on what little prior subject-matter knowledge of y there is, which I then (b) turn into a linear combination of a parsimonious subset of all 350 predictors (say e.g. 10 of them) and regress y on that subset in a more-or-less vanilla manner and finally (c) evaluate the performance of that model in some acceptable way (R^2; stasitstical significance of predictor betas; accuracy on a testing set of a model fit on a training set; etc.) and write up the result (e.g. "we found that variable A was a statistically significant predictor of y, but that variables B and C were not" -- this would be the standard/traditional social science approach, or maybe "after training the model on a randomly selection proportion of the data, its performance on a testing set of "new" data was ____" -- this would be the more modern approach).
(a) forgo any prior model and (b) use a sort of 'kitchen sink plus regularization' approach in which y is regressed onto all 350 predictors and a regularization / variable selection method like Elastic Net is used (in combination with k-fold CV to select optimal tuning parameters) to "find" a model - its own subset of predictors - which maximizes prediction accuracy.
Here's my dilemma, then: what if I do both of these and find that (2) is not only better at predicting y than my own hypothesized (1), but that the predictors selected by (2) are different (partially or entirely) from the ones I selected a priori? What does that mean for the phenomenon as it exists out there in nature -- is Elastic Net saying something about "nature" at all? Would I have to throw out my own model, or figure out a way to account for any "new" variables that I hadn't hypothesized would matter but that the regularization method is telling do matter?
Would it make any difference if, in Approach (2), instead of throwing all 350 variables into the model, I threw the subset of predictors I used in Approach (1) -- or would that be changing the whole question?
This question is closely related to another asked on CV [1, 2], but it isn't quite the same, I don't think. If this has been addressed in the past, I apologize for having overlooked it!