In "To Explain or to Predict?", Pr. Galit Shmueli said that sometimes a less true model can predict better than a truer model. Why is it so? When will it happen? How does it happen? Is explanation a trade-off for prediction in models?
$\begingroup$
$\endgroup$
3
-
3$\begingroup$ Have you read Shmueli's article? From reading the abstract, it seems that his article is aimed at precisely answering these questions. Are there particular arguments he makes that you have questions about? $\endgroup$– Peter EllisCommented Feb 10, 2012 at 5:32
-
$\begingroup$ Note that the full text is available via Google Scholar: scholar.google.com/… $\endgroup$– Jack TannerCommented Feb 10, 2012 at 5:55
-
$\begingroup$ It is by now means clear how to define whether one model is "truer" than another, and any answer will have to rely on how exactly this is defined. $\endgroup$– Christian HennigCommented Jul 16, 2022 at 13:12
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
6
I believe that this is one of the most counter-intuitive aspects of statistics; it's just really difficult to wrap your head around. The key notion here is the idea of the bias-variance tradeoff. It has been discussed in several places on CV, and you may want to check out some of the other answers, for example here or here, and I discussed it before here. Setting mine aside, the other two are quite good and well worth your time.
I will try to give a quick sense of the idea. Let me first define some terms. To start with, what Shmueli means by "true" model is the actual data generating process; the closer your estimated model is to the real data generating process, the truer it is. For instance, if $\beta_1=.5$, and one model fit yields $\hat{\beta}_1=.6$, that's truer than another fit that yields $\hat{\beta}_1=.7$. On the other hand, predicting better means getting your $\hat{y}$'s as close as possible to the actual $y$'s, especially for out-of-sample data. Notice the differences in goals here (because that's crucial to understanding the issue): getting $\hat\beta$'s as close as possible vs. getting $\hat{y}$'s as close as possible. So Shmueli's point is that sometimes your $\hat{y}$'s can be closer to the actual $y$'s when your $\hat\beta$'s are estimated by a process that, on average, yields values a little further from the true $\beta$'s. Now, how is that possible?
The key is that there is variance associated with parameters estimated from sample data. For a given sample, sometimes the maximum likelihood estimate happens to be further from the true value and sometimes closer. It is quite possible to have a situation where the variance of the sampling distribution of a parameter estimate is so large that $\hat\beta$'s routinely bounce so far out around their true value that they are not worth much. The thing to remember here is that classical statistics are based on what are called the 'best linear unbiased estimators', that is, the estimators that have the lowest variance of all the unbiased estimators. However, there can be other ways of attempting to get an estimate that are not unbiased. Typically, these have been developed within machine learning (a subfield of computer science). It is possible in some cases to have an estimate that doesn't tend to bounce as far from the true value, even though the sampling distribution of that estimate is not centered on the true value (i.e., it is biased). Given all of this, what matters for the accuracy of your predictions is how the inaccuracy due to the induced bias trades off vis-a-vis the inaccuracy induced by the high variance of the BLUE parameter estimate (hence the name). Specifically, if the inaccuracy due to higher variance is greater than the inaccuracy due to the bias, the less true model will give the better predictions.
answered Feb 10, 2012 at 5:54
-
2$\begingroup$ +1: I think you brought it all together in one place. Excellent! $\endgroup$– WayneCommented Feb 10, 2012 at 14:48
-
1$\begingroup$ I'd also add that I've run into a similar, but different problem: I create a simplified model and it does reasonably well. Then I add in variables that "really should be in there because it's more physically realistic (truer)" and my results are worse. Not because of bias-variance, but because my model is literally poorer: I've introduced collinearity, my new variables are measured more poorly than the easy variables I started with, I've introduced "real world" variables that are my theory rather than actually real, etc. I know it's not what Shmueli's talking about and it's not as sexy, but... $\endgroup$– WayneCommented Feb 10, 2012 at 14:55
-
1$\begingroup$ @gung Thank you for your great answer. I think the bias-variance tradeoff is similar to accuracy and precision in measurement. However, true model is never known in the real world. Thus, we can only define a useful model by its predictive ability but we can never define a true model. Does this mean that prediction is a fairer judge than explanation $\endgroup$– KuJCommented Feb 10, 2012 at 15:42
-
$\begingroup$ @Wayne, +1. I think that's an excellent example, and I think it's actually related to these issues. $\endgroup$ Commented Feb 10, 2012 at 16:24
-
$\begingroup$ @Jinn-YuhGuh, we don't know the true model in the real world, but it's OK. If your goal is to get ever better approximations of the true model over the course of the research process (which is often a good description of what's going on), then explanatory modeling is best. It really depends on what your goals are. To get a fuller sense of these issues, in addition to the link to the paper above, Shmueli has a flash version of a full talk she gave on this topic: galitshmueli.com/explain-predict & there was a related question on CV: stats.stackexchange.com/questions/18896 $\endgroup$ Commented Feb 10, 2012 at 16:39