3
$\begingroup$

Is it possible for a statistical model to have explanatory power but no predictive power?

$\endgroup$
5
$\begingroup$

I'm going to offer the dissenting opinion on @Sympa's answer, and go with, No.

It's not really well defined what you mean by explanatory power, so I'll offer a definition. A model has explanatory power if it can be used to draw qualitative inferences about the underlying statistical process. A qualitative inference could be, for example, the sign or direction of an association between two variables of interest (a sign or direction of casuation is a separate issue which I am setting aside for the moment).

A statistical model is an approximation to an unseen process of interest to the researcher. If a model has no predictive power, i.e. no ability to correctly anticipate unseen data, then fundamentally the model is not representing the underlying statistical process being studied. If this is so, then any qualitative inferences drawn from said model can not be confidently generalized to inferences about the underlying process.

So, I would say, some level of predictive power should be a necessary condition for statistical explanation.

$\endgroup$
  • $\begingroup$ I'm not sure this is always true. Using a simple low dimension example to illustrate, generate two lists of 20 data points: group A: 50, 60, 70,..., 240, and group B: 90, 100, 110,..., 280. Calculating the means and stnd errors yields mean_A = 145 and mean_B = 185, with stnd errors of 12.9 and two-tailed t-test p-value of .039. There's a statistically significant difference between group A and group B - we can explain something important about the data. $\endgroup$ – RobertF Apr 16 at 16:11
  • $\begingroup$ However, if you lump the data pts together and then attempt to predict the outcome of group A or B, say with a decision tree or SVM, at best you'll correctly identify the group for only 24 out of the 40 data pts, an accuracy of only 0.60. $\endgroup$ – RobertF Apr 16 at 16:14
  • $\begingroup$ @RobertF Fair enough, but I would counter with the tired and true "why accuracy"? If you use a probabilistic model, like a logistic regression or gradient booster (or just have your tree predict probabilities), you can certainly make predictions for the probability of group membership. To me, you can certainly gain some predictive power in this situation. $\endgroup$ – Matthew Drury Apr 16 at 19:15
  • $\begingroup$ I don't think PPV or sensitivity measures would do much better. For example, predicting group A membership for the 4 lowest values and a random pick of half of the values with overlapping A/B membership would yield average PPV and sensitivity of 0.60. That's still better than PPV and sensitivity of 0.50 for guessing A or B at random, but not by much. $\endgroup$ – RobertF Apr 18 at 17:23
  • $\begingroup$ Yes, but all of those are predicated on "predictive power" being measured according to hard group assignment. Instead, I would argue it is better assessed probabilistically. I can say in your example that the posterior-probability of membership in one group is clearly higher than another conditional on the data. That's what predictive power means (to me). The ability to confidently make hard classifications is a seperate issue. We may be using terminology in different ways. $\endgroup$ – Matthew Drury Apr 18 at 19:13
3
$\begingroup$

Yes, I think it happens quite often. Let's say you could have a model estimating GDP growth (as the dependent variable) using two independent variables: change in national home prices and change in the S&P 500. That model may have good explanatory power. It may have a high R Square. The two independent variables may have the correct positive sign and be very statistically significant. And, most important the model's explanatory power could be very well supported by economic theory (Hyman Minsky's credit cycles associated with home prices, etc.). But, when you test this model using Hold Out sample it may not perform that well. It may not be predictive. In other words, change in home prices and the stock market can explain a significant percentage of economic growth (after all that is one of the meanings of R Square). But, such a model may be mediocre at actually predicting the future path of the economy. That's almost a summary of the quasi-tragedy of a large body of econometrics models. Many are or appear to be very good at "explaining", so far I am not sure we have found many or any that are good at "predicting."

$\endgroup$
1
$\begingroup$

The correct answer is yes. A statistical model can have explanatory value without being a good predictor. But it depends upon what you are trying to predict.

The classic example? Sex differences in body weights.

Studies show that on average males are heavier than females. I can model the effect, estimate the difference and attach estimates of uncertainty to that estimate.

But can I use it for prediction?

Well it depends upon what I'm trying to predict.

  1. If I'm trying to predict the sex of an individual based on their body weight then my statistical model is unlikely to perform much better than chance.

  2. If I'm trying to separate two groups of same-sex individuals into the male and female groups then I'm in good shape. Providing we have an unbiased sample of a reasonable size then I should be able to do this with some confidence.

$\endgroup$
  • 1
    $\begingroup$ "If I'm trying to separate two groups of same-sex individuals into the male and female groups" - it's unclear what you mean by this. Do you mean you have a group of males and a group of females, and you are trying to identify which is which? $\endgroup$ – jbowman Feb 5 '18 at 16:41
  • 1
    $\begingroup$ That's absolutely correct - I have a group of males and a group of females, and I am trying to identify which is which. Thanks for clarifying. $\endgroup$ – dennislendrem Feb 15 '18 at 17:53
  • $\begingroup$ Wouldn't this model be able to make good distinctions between the posterior probability an individual is male and female? Are you taking "prediction" to mean "class assignment"? $\endgroup$ – Matthew Drury Apr 16 at 19:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.