Is prediction the 'golden criterion' to judge the ability of statisticans? I was reading Faraway's textbook linear models with R (1st edition) last weekend. Faraway had a chapter called "Statistical Strategy and Model Uncertainty". He described (page 158) that he artificially generated some data using a very complicated model, then he asked his students to model the data and compare the students' predicted results vs read results. Unfortunately, most students over-fitted the testing data and gave predicted values totally off the mark. To explain this phenomenon, he wrote something very impressive to me:
"
The reason the models were so different was that students applied the various methods
in different orders. Some did variable selection before transformation and others, the
reverse. Some repeated a method after the model was changed and others did not. I went
over the strategies that several of the students used and could not find anything clearly
wrong with what they had done. One student made a mistake in computing his or her
predicted values, but there was nothing obviously wrong in the remainder. The
performance on this assignment did not show any relationship with that in the exams.
"
I was educated that the model prediction accuracy is the 'golden criterion' for us to select the best model performance. If I am not mistaken, this is also the popular method used in Kaggle competitions. But here Faraway observed something of a different nature, that the model prediction performance could have nothing to do with the ability of the statistican involved. In other words, whether we can build the best model in terms of predictive power is not really determined by how experienced we are. Instead it is determined by a huge 'model uncertainty' (blind luck?). My question is: is this true in real life data analysis as well? Or was I confused with something very basic? Because if this is true, then the implication to real data analysis is immense: without knowing the "real model" behind the data, there is no essential difference between the work done by experienced/inexperienced statisticans: both are just wild guesses in front of the training data available. 
 A: I asked the professor in my department on this. He said frankly he was not surprised about it at all. He suggested the following way to look at this: what Faraway did was only a one time experiment, and it is not surprising that the results appeared to have no correlation with the final grades. But if Faraway repeat his 'experiment' 100 times with the same group of students, he is sure that the students learned statistics better would be performing well, similar as the confidence interval. So in his opinion experience does matter, it is just a one time social experiment could not show it because of the model uncertainty. 
A: The students' models were almost all overfit.  With n data points, one can always fit a perfect polynomial of order n-1.  Such a model is overdue leaving nothing to random error.   It appears the students have made similar overfittng errors, but presumably with different functions.  
Overfitting is an error that should only be made by students.   And this suggests experience and education are necessary qualifications for modelling. 
