I'm reading Computer Age Statistical Inference by Efron and Hastie, two statisticians I have a lot of respect for.

Section 12.3 discusses Mallows' $C_{p}$, Akaike's information criteria (AIC), and Stein's unbiased risk estimator (SURE), which the authors collectively call "Covariance Penalties", for estimating prediction error. They conclude this section with the following recommendation.

Covariance penalty estimates, when believable parametric models are available, should be preferred to cross-validation.

Is this understanding conventional? I have always preferred cross-validated prediction error as a primary measure, and I throw in a covariance penalty measure for additional insight if I'm reasonably confident in my modeling assumptions. I guess my key gripe here is the authors' use of the word believable. Before reading this passage, my understanding was that cross-validated prediction error should be preferred when it is available because it does not require the additional assumption of a statistical model. Even when the model seems believable, it is unlikely that it holds in every detail, and for that reason cross-validated prediction error is more reliable.

The authors give some justification for this claim by demonstrating that a poorly chosen test set is not a good measure of prediction error. This isn't too surprising. I'm interested in hearing others' opinion on the topic and their justification.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.