In one of the paper I am writing, I am looking at cross-validation on bootstrap samples. I wrote the following explanation.One of the reviewer wrote that he didn't understand where is the correlation. I don't know how to explain better. So I was wondering if someone has a reference I could cite about this issue when mixing cross-validation and bootstrap. Thanks!

In the context of cross-validation, we have to be careful. Cross-validation divides the data in two parts, the training set and the test set. For the original data, every individual observation appears once in the dataset (we assume all the (X, Y ) p + 1 tuples are distinct.) Hence there is no possibility that an individual is used in the training and the test set, a potential source of positive bias. This is not the case for a bootstrapped dataset — there will almost certainly be ties. If we let some of these tied values go to the training set, and some to the validation set, we will have artificially created (additional) correlation. To avoid this, we operate at the level of observation weights. In the original sample, each observation has weight 1/n. In a bootstrapped sample, these weights will be of the form k/n, for k = 0, 1, 2, . . .. When we cross-validate a bootstrap sample, we randomly divide the original observations into the two groups, but their bootstrap weights go with them (including the 0s).

  • 1
    $\begingroup$ Maybe the discussion in the Elements of Statistical Learning of out-of-bootstrap error estimation is helpful? (Chapter 7.11 in the 2009 edition) $\endgroup$ – cbeleites unhappy with SX Mar 11 '14 at 20:51
  • 1
    $\begingroup$ I understand what you mean (and I think it makes sense), but I must also say that your description is quite confusing. I don't know of a suitable reference, but you should definitely try to improve your wording. $\endgroup$ – amoeba Mar 11 '14 at 22:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.