3
$\begingroup$

I have a background in statistics (for social science), but I am confused about the ways in which Data Science textbooks (in particular, An Introduction to Statistical Learning and Practical Statistics for Data Scientists) describe the relationship between inference, prediction, & model fit.

How does whether you want your model to (1) predict an outcome or (2) infer relationships between X and Y effect how you go about fitting the model? Is it simply a matter of whether you use validation and holdout sets (for prediction) instead of fitting a model to all data (for inference)? Based on my reading of the texts, it is more complicated than that. For example: Do we need to minimize a different statistic when fitting an inferential model vs a predictive model? Do we typically include a different number of predictors in an inferential vs predictive model? Etc.

$\endgroup$
1

2 Answers 2

3
$\begingroup$

The best answer I've come across can be found in To Explain or To Predict? (Shmueli 2010). In the article, the author breaks down the choices that an analyst makes (the "two modeling paths") depending on whether their goal is prediction or explanation (i.e., inference). This is not a complete list of considerations when building models, but it does provide a basis for understanding how a predictive model and an explanatory model may differ given the same data.

In short, explanatory models focus on minimizing bias (i.e., the difference between the expected and true value of an estimator) in order to more accurately represent the underlying relationship between X and Y, whereas predictive models minimize both bias and estimation variance (std. error). Additionally, explanatory problems require that the model's coefficients are interpretable (for somewhat obvious reseasons), whereas predictive problems often sacrifice interpretability for greater predictive power (e.g., using 'black box' algorithmic and nonparametric models).

More to my initial question: Explanatory models are evaluated using 'goodness of fit' tests (e.g., R-squared, Mallow's Cp, etc.) and other model diagnostics (e.g., residual analysis) that "[measure] the strength of the relationship indicated by f-hat" (p 16). Evaluating predictive models involves comparing the performance of the model on the training vs the holdout datasets. The predictive model should minimize the error on the test set (or else the model may be over-fitted). Additionally, explanatory models may have to worry about identifying sources of endogeneity, collinearity, etc., which increase bias in the model. These concerns are minimized for predictive models.

Finally, theoretical considerations (e.g., expected causal relationships) play a role in model evaluation, though more so for explanatory models. For example, "a researcher might choose to retain a causal covariate which has a strong theoretical justification even if is statistically insignificant" (p 17). This would never be done in a predictive model because it would decrease its predictive power. Similarly, because interpribility is paramount in explanatory models, these may include additional variables that do nothing for predictive power (e.g., main and interaction terms).

All in all, though it is obviously important to know what sort of problem you have before starting your analysis, explanation models and prediction models are two sides of the same coin. There is always a tension between minimizing variance and bias and analysts need to consider what techniques minimize the stat/s that will create the best model.


Note: f refers to the model function, where F refers to an underlying function that describes the true relationship between X and Y. In other words, the true relationship between X and Y is:

Y = F(X),

whereas the statistical model is:

E(Y) = f(X),

where X and Y are operationalizations of X and Y.

$\endgroup$
3
  • $\begingroup$ (+1) Small technical gripe: "A model should fit the test data as well as it did the training data", I think that should be "Minimize the error on the test set.". This can be very different than the training and testing data being the same/similar, as in random forests. $\endgroup$ Mar 30, 2020 at 15:25
  • $\begingroup$ You may be interested in this question, where Shmueli herself weighs in: stats.stackexchange.com/questions/342360/… $\endgroup$ Mar 30, 2020 at 15:26
  • $\begingroup$ @MatthewDrury Thank you. I edited my answer to reflect this. $\endgroup$
    – peterlista
    Mar 30, 2020 at 15:33
2
$\begingroup$

Ok, interesting question.

I think one might be tempted to say the procedures are the same regardless of if you are doing prediction or inference. Minimizing a least squares criterion does not depend on your intentions.

What does change is your perspective on the problem. Whereas prediction only more often than not cares about the conditional mean of the outcome (that is $\mathbb{E}(y \vert x)$), inference cares about the conditional distribution.

EDIT: I will admit that this is my perception on the distinction of inference v. prediction and that others may differ. As Glen_b has pointed out, bootstrapping prediction intervals can be considered part of prediction, though it does not concern point estimates. It is my own opinion that Inference v. Prediction is a question of Distributions vs. Point Estimates, and if I were forced to a weaker position then I would say Distributions vs. Point Estimates is a good first approximation to Inference v. Prediction. In making this distinction, I'm looking for a distinction which is 80% as correct to the true distinction and requires 20% of the time to convey to someone as the true distinction. I would encourage you to take a holistic perspective on the distinction by considering other takes rather only mine in isolation.

The choice of likelihood affects confidence intervals, p values, and all the other inferential statistics we care about. So although we can minimize squared errors on both cases, in inference we ask should we minimize squared errors, or some other loss function? I see an ambivalence in data science (for better or worse) towards the data generating process. Everything has either a gaussian likeihood (sum of squared errors loss) or a binomial likelihood (cross entropy loss), even for things which are very clearly not gaussian or binomial (see a discussion about likelihoods here for instance).

As to your point about holdout sets, that concerns model validation which in my experience does differ between predictive and inferential models.

$\endgroup$
6
  • $\begingroup$ "Whereas prediction only cares about the conditional mean of the outcome" --- I find this quite surprising; I often calculate prediction intervals. $\endgroup$
    – Glen_b
    Mar 30, 2020 at 14:31
  • $\begingroup$ @Glen_b-ReinstateMonica I'm marginalizing over all practitioners. Also, not all prediction methods are amenable to the creation of prediction intervals. You would have to specify a conditional distirbution, in which case I think you are heading towards inference-land, but that could very well be semantics. $\endgroup$ Mar 30, 2020 at 14:37
  • $\begingroup$ Bootstrap prediction intervals don't require one to specify a conditional distribution $\endgroup$
    – Glen_b
    Mar 30, 2020 at 14:40
  • $\begingroup$ @Glen_b-ReinstateMonica, sure but the bootstrap is meant to approximate the sampling distribution of whatever statistic you're bootstrapping, no? So we're back to talking about distributions. Again, I think is is coming down to semantics. Very rarely in my experience have arguments on semantics (is this or that inference or prediction) yielded something useful, so you'll have to excuse me if I don't find this topic compelling. $\endgroup$ Mar 30, 2020 at 14:43
  • 1
    $\begingroup$ @Glen_b-ReinstateMonica There comes a point where semantics are more of a barrier. By trying to accommodate everyone's perceptions on the distinction between inference and prediction, answers become less clear rather than more clear, in my opinion. I don't doubt that some practitioners focus on more than just point estimates of predictions, however I would be remiss to say that all practitioners do. I have edited my answer in hopes you find it more amenable. $\endgroup$ Mar 30, 2020 at 15:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.