I know there are many related questions to this but none are exactly to the point I want to ask , My question is in terms of simple linear regression,

This statement(In bold)from the book An Introduction to Statistical Learning

In Leave-One-Out Cross validation the statistical learning method is fit on the n − 1 training observations, and a prediction is made for the excluded observation, using its value x1. Since (x1, y1) was not used in the fitting process, MSE1 = provides an approximately unbiased estimate for the test error. But even though MSE1 is unbiased for the test error, it is a poor estimate because it is highly variable, since it is based upon a single observation (x1, y1).

So my questions are.

  1. How can I tell whether any method/model provides an unbiased estimate for the TEST ERROR?(I know well about unbiased estimates for parameters).

  2. How does the above apply to this specific case of Leave-One-Out Cross validation ?

  • 1
    $\begingroup$ See statweb.stanford.edu/~tibs/ElemStatLearn/printings/… section 7.10 Cross-Validation and the discussion of "how the performance of the learning method varies with the size of the training set" (page 262 of the PDF, page 243 of the book) $\endgroup$ – Adrian Mar 8 '17 at 15:09
  • $\begingroup$ Have you looked at the book by Bradley Efron and Rob Tibshirani on the bootstrap? $\endgroup$ – Michael R. Chernick Mar 8 '17 at 16:44
  • $\begingroup$ No , I haven't ., what is the name of the book? how is it related to my question? $\endgroup$ – GeneX Mar 8 '17 at 16:50
  • $\begingroup$ In "On estimation of characters obtained in statistical procedure of recognition", Luntz and Brailovsky showed that Leave-One-Out Cross validation is almost an unbiased estimator. It is a paper from 1969, written in Russian, and I couldn't find it on the web. You can see here a description of the result. $\endgroup$ – DaL Mar 9 '17 at 9:09
  • $\begingroup$ @DaL, you can see this in plenty of writings in English. Take a look at Tony Lachenbruch's original work on this back in 1967. I explicitly discuss it in my bootstrap book when discussing estimates of classification error. For example read page 35 in my book "An Introduction to Bootstrap Methods with Applications to R" published by Wiley in 2011 & coauthored with Robert LaBudde. $\endgroup$ – Michael R. Chernick Sep 16 '19 at 19:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.