I have a project where I need to 1) perform inference: understand the role of predictors on the response through models (with my data I need to choose a model where I can reasonably assess how changes in x affect y) and 2) perform prediction: get the best prediction on future y's (with my data how well can I build a model to use a future data set to accurately predict unknown future y's).

My issue is that I feel like these overlap in a substantial way or at least, I'd like to use them in a complementary way.

For inference, I understand simple models are generally preferred, where we are interested in using goodness of fit measures to assess model assumptions or more generally, to compare competing models (something as simple as testing the significance on a single predictor). But, those goodness of fit tests often rely on substantial distributional assumptions on a statistic (which become worse with more complex models, making p-values highly unreliable). Also, traditional goodness of fits tests have limited use/highly prectionary use in comparison of competing non-nested models/or models where the response variable has been transformed.

This brings me to my mini question: So, in comparing a few competing models, is this a place where a prediction technique like cross validation (on the whole data set) could be used to get a reasonable sense for goodness of fit, especially since a lot of goodness of fit stats already penalize for increased parameters (adjusted $R^2$ etc.) and the risk of overfitting?

For prediction, methods like elastic net and random forests (the latter albeit non parametric), once tuned, provide us with a sense of the role of predictors in the prediction performance of the model e.g. shrinkage and variable selection and variable importance respectively. So, they give us a sense of what's important to make future $\hat{y}$ look like true future $y$. Of course, these models are tuned over only a subset of the data and then assessed on their ability to predict, they try to reduce variance as well as bias, and the effect of each variable on the response is not interpretable.

So, my major question is ... is there a nice way to combine the aims of prediction and inference? Does anyone have good suggestions?

I know it's customary to give a possible solution to the question asked so below is my attempt!

One thought that I have : might someone look at a few simple models for inference on the whole data set using prior knowledge of what models are reasonable, get a sense of how the predictors affect the response (eg. a one unit increase in blah, leads to...), and compare these using cross validation on the whole data set (say a random intercept model vs. one without). Thus, they'd avoid problematic distributional assumptions for goodness of fit tests and have an interpretable value for comparison.

Then, for prediction, they could include more complex models (which actually can achieve the simple models above/or close to if the hyperparameters turn out to be tuned as such) and then run cv on training data and evaluating on the test data. All the while, they could note what predictors seem to be important (those selected or those maximally reducing RSS)?

This was a suggestion to a similar question: How can I compare my model to a technically invalid model?

I've looked at: Inference, Prediction, & Model Fit?, What is the relation between causal inference and prediction?, and How to choose a ML model when the goal is both reasonable prediction AND inference? and have posted this related question,AIC Debate Once Again...Comparison of Likelihoods Across Distributions

  • $\begingroup$ You are right: inference and prediction can be (and often are) related, but they are distinct in that when we predict, we want our model to perform well with unknown data. Your proposed solution can work, as long as you note your assumptions (these applies to models for any purpose, but particularly for inference). Regarding cross-validation to evaluate inferences made by simpler models, it could be useful, but cross-validation is generally used to evaluate the predictive ability of a model's assumptions, validating assumptions your model makes is more important. $\endgroup$ – atirvine88 Apr 28 at 19:37
  • $\begingroup$ Thank you! My issue is that outside of linear regression, goodness of fit approaches can become hairy (and are debated). Comparison statistics between competing models either require nested models (often with strong distributional assumptions) or unreliable statistics like AIC (which can't deal with transformed data or may not include the right constants for comparison). What if we're really unsure of how to model the response? Maybe a normal model in a crazy world really outperforms a Poisson model (mean = variance) on count data via cv. Can we not interpret the predictors in the normal fit? $\endgroup$ – Winston Apr 28 at 20:29
  • $\begingroup$ At the end of the day, in the data you're working with, inclusion of predictors will still help explain variation in the response (even if marginally)...what's the purpose of goodness of fit if only to tell you that you haven't explained enough of it assuming some stuff... Edit: of course with some reasonable understanding of what the response variable could be distributed as. $\endgroup$ – Winston Apr 29 at 0:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.