# Is prediction the 'golden criterion' to judge the ability of statisticans?

I was reading Faraway's textbook linear models with R (1st edition) last weekend. Faraway had a chapter called "Statistical Strategy and Model Uncertainty". He described (page 158) that he artificially generated some data using a very complicated model, then he asked his students to model the data and compare the students' predicted results vs read results. Unfortunately, most students over-fitted the testing data and gave predicted values totally off the mark. To explain this phenomenon, he wrote something very impressive to me:

" The reason the models were so different was that students applied the various methods in different orders. Some did variable selection before transformation and others, the reverse. Some repeated a method after the model was changed and others did not. I went over the strategies that several of the students used and could not find anything clearly wrong with what they had done. One student made a mistake in computing his or her predicted values, but there was nothing obviously wrong in the remainder. The performance on this assignment did not show any relationship with that in the exams. "

I was educated that the model prediction accuracy is the 'golden criterion' for us to select the best model performance. If I am not mistaken, this is also the popular method used in Kaggle competitions. But here Faraway observed something of a different nature, that the model prediction performance could have nothing to do with the ability of the statistican involved. In other words, whether we can build the best model in terms of predictive power is not really determined by how experienced we are. Instead it is determined by a huge 'model uncertainty' (blind luck?). My question is: is this true in real life data analysis as well? Or was I confused with something very basic? Because if this is true, then the implication to real data analysis is immense: without knowing the "real model" behind the data, there is no essential difference between the work done by experienced/inexperienced statisticans: both are just wild guesses in front of the training data available.

• +1 nice question. To offer another angle, say one of the analysts knows the true mode - then her predictions may be bad too! So even with knowing the real model, you'd see this. Important may be Haggerty and Srivinasans 1991 observation in Psychometrika that "the practice [...] of concluding that a model with higher predictive accuracy is "truer" is not a valid inference".
– Momo
Commented Dec 8, 2015 at 10:37
• I've not looked at the book yet, but "variable selection" & "transformation" already ring warning bells. See Algorithms for automatic model selection & Nature of the Relationship between Predictors and Dependent in Regression. I also wouldn't conflate the exam performance of Statistics students with the real-work ability of statisticians. Commented Dec 8, 2015 at 11:05
• This information given by Faraway seems awfully anecdotal to be used as the basis for a sweeping general principle about the field of statistics. I wouldn't want to construct a model about predictive modeling based on such non-reproducible examples. It's also possible that they were, wittingly or not, cherry-picked. Commented Dec 8, 2015 at 11:20
• One logically valid conclusion that can be derived from this anecdote is that none of Faraway's students had (yet) acquired skills needed to perform well on his prediction test. It is difficult to make any connection at all between that result and your speculations about how experienced statisticians might perform.
– whuber
Commented Dec 8, 2015 at 14:25
• @whuber: I do not think that way. I agree 28 students are a bit small, but I do think this real observation has some serious implications. If Faraway made the real model, and he went over with several students' work, could not find any serious mistake, yet the predictions are way-off from what they should be. Then this says something about the 'model uncertainty' being involved, that one at least need the work done by a separate analyst to compare the differences, no matter how 'experienced' the original analyst is. I think this is quite alarming to me. Commented Dec 8, 2015 at 20:26