I have a project where I need to 1) perform inference: understand the role of predictors on the response through models (with my data I need to choose a model where I can reasonably assess how changes in x affect y) and 2) perform prediction: get the best prediction on future y's (with my data how well can I build a model to use a future data set to accurately predict unknown future y's).
My issue is that I feel like these overlap in a substantial way or at least, I'd like to use them in a complementary way.
For inference, I understand simple models are generally preferred, where we are interested in using goodness of fit measures to assess model assumptions or more generally, to compare competing models (something as simple as testing the significance on a single predictor). But, those goodness of fit tests often rely on substantial distributional assumptions on a statistic (which become worse with more complex models, making p-values highly unreliable). Also, traditional goodness of fits tests have limited use/highly prectionary use in comparison of competing non-nested models/or models where the response variable has been transformed.
This brings me to my mini question: So, in comparing a few competing models, is this a place where a prediction technique like cross validation (on the whole data set) could be used to get a reasonable sense for goodness of fit, especially since a lot of goodness of fit stats already penalize for increased parameters (adjusted $R^2$ etc.) and the risk of overfitting?
For prediction, methods like elastic net and random forests (the latter albeit non parametric), once tuned, provide us with a sense of the role of predictors in the prediction performance of the model e.g. shrinkage and variable selection and variable importance respectively. So, they give us a sense of what's important to make future $\hat{y}$ look like true future $y$. Of course, these models are tuned over only a subset of the data and then assessed on their ability to predict, they try to reduce variance as well as bias, and the effect of each variable on the response is not interpretable.
So, my major question is ... is there a nice way to combine the aims of prediction and inference? Does anyone have good suggestions?
I know it's customary to give a possible solution to the question asked so below is my attempt!
One thought that I have : might someone look at a few simple models for inference on the whole data set using prior knowledge of what models are reasonable, get a sense of how the predictors affect the response (eg. a one unit increase in blah, leads to...), and compare these using cross validation on the whole data set (say a random intercept model vs. one without). Thus, they'd avoid problematic distributional assumptions for goodness of fit tests and have an interpretable value for comparison.
Then, for prediction, they could include more complex models (which actually can achieve the simple models above/or close to if the hyperparameters turn out to be tuned as such) and then run cv on training data and evaluating on the test data. All the while, they could note what predictors seem to be important (those selected or those maximally reducing RSS)?
This was a suggestion to a similar question: How can I compare my model to a technically invalid model?
I've looked at: Inference, Prediction, & Model Fit?, What is the relation between causal inference and prediction?, and How to choose a ML model when the goal is both reasonable prediction AND inference? and have posted this related question,AIC Debate Once Again...Comparison of Likelihoods Across Distributions
