Inference, Prediction, & Model Fit?

Question

I have a background in statistics (for social science), but I am confused about the ways in which Data Science textbooks (in particular, An Introduction to Statistical Learning and Practical Statistics for Data Scientists) describe the relationship between inference, prediction, & model fit.

How does whether you want your model to (1) predict an outcome or (2) infer relationships between X and Y effect how you go about fitting the model? Is it simply a matter of whether you use validation and holdout sets (for prediction) instead of fitting a model to all data (for inference)? Based on my reading of the texts, it is more complicated than that. For example: Do we need to minimize a different statistic when fitting an inferential model vs a predictive model? Do we typically include a different number of predictors in an inferential vs predictive model? Etc.

Answer 1

Ok, interesting question.

I think one might be tempted to say the procedures are the same regardless of if you are doing prediction or inference. Minimizing a least squares criterion does not depend on your intentions.

What does change is your perspective on the problem. Whereas prediction only more often than not cares about the conditional mean of the outcome (that is $\mathbb{E}(y \vert x)$), inference cares about the conditional distribution.

EDIT: I will admit that this is my perception on the distinction of inference v. prediction and that others may differ. As Glen_b has pointed out, bootstrapping prediction intervals can be considered part of prediction, though it does not concern point estimates. It is my own opinion that Inference v. Prediction is a question of Distributions vs. Point Estimates, and if I were forced to a weaker position then I would say Distributions vs. Point Estimates is a good first approximation to Inference v. Prediction. In making this distinction, I'm looking for a distinction which is 80% as correct to the true distinction and requires 20% of the time to convey to someone as the true distinction. I would encourage you to take a holistic perspective on the distinction by considering other takes rather only mine in isolation.

The choice of likelihood affects confidence intervals, p values, and all the other inferential statistics we care about. So although we can minimize squared errors on both cases, in inference we ask should we minimize squared errors, or some other loss function? I see an ambivalence in data science (for better or worse) towards the data generating process. Everything has either a gaussian likeihood (sum of squared errors loss) or a binomial likelihood (cross entropy loss), even for things which are very clearly not gaussian or binomial (see a discussion about likelihoods here for instance).

As to your point about holdout sets, that concerns model validation which in my experience does differ between predictive and inferential models.

Answer 2

The best answer I've come across can be found in To Explain or To Predict? (Shmueli 2010). In the article, the author breaks down the choices that an analyst makes (the "two modeling paths") depending on whether their goal is prediction or explanation (i.e., inference). This is not a complete list of considerations when building models, but it does provide a basis for understanding how a predictive model and an explanatory model may differ given the same data.

In short, explanatory models focus on minimizing bias (i.e., the difference between the expected and true value of an estimator) in order to more accurately represent the underlying relationship between X and Y, whereas predictive models minimize both bias and estimation variance (std. error). Additionally, explanatory problems require that the model's coefficients are interpretable (for somewhat obvious reseasons), whereas predictive problems often sacrifice interpretability for greater predictive power (e.g., using 'black box' algorithmic and nonparametric models).

More to my initial question: Explanatory models are evaluated using 'goodness of fit' tests (e.g., R-squared, Mallow's Cp, etc.) and other model diagnostics (e.g., residual analysis) that "[measure] the strength of the relationship indicated by f-hat" (p 16). Evaluating predictive models involves comparing the performance of the model on the training vs the holdout datasets. The predictive model should minimize the error on the test set (or else the model may be over-fitted). Additionally, explanatory models may have to worry about identifying sources of endogeneity, collinearity, etc., which increase bias in the model. These concerns are minimized for predictive models.

Finally, theoretical considerations (e.g., expected causal relationships) play a role in model evaluation, though more so for explanatory models. For example, "a researcher might choose to retain a causal covariate which has a strong theoretical justification even if is statistically insignificant" (p 17). This would never be done in a predictive model because it would decrease its predictive power. Similarly, because interpribility is paramount in explanatory models, these may include additional variables that do nothing for predictive power (e.g., main and interaction terms).

All in all, though it is obviously important to know what sort of problem you have before starting your analysis, explanation models and prediction models are two sides of the same coin. There is always a tension between minimizing variance and bias and analysts need to consider what techniques minimize the stat/s that will create the best model.

---

Note: f refers to the model function, where F refers to an underlying function that describes the true relationship between X and Y. In other words, the true relationship between X and Y is:

Y = F(X),

whereas the statistical model is:

E(Y) = f(X),

where X and Y are operationalizations of X and Y.