In one of the paper I am writing, I am looking at cross-validation on bootstrap samples. I wrote the following explanation.One of the reviewer wrote that he didn't understand where is the correlation. I don't know how to explain better. So I was wondering if someone has a reference I could cite about this issue when mixing cross-validation and bootstrap. Thanks!
In the context of cross-validation, we have to be careful. Cross-validation divides the data in two parts, the training set and the test set. For the original data, every individual observation appears once in the dataset (we assume all the (X, Y ) p + 1 tuples are distinct.) Hence there is no possibility that an individual is used in the training and the test set, a potential source of positive bias. This is not the case for a bootstrapped dataset — there will almost certainly be ties. If we let some of these tied values go to the training set, and some to the validation set, we will have artificially created (additional) correlation. To avoid this, we operate at the level of observation weights. In the original sample, each observation has weight 1/n. In a bootstrapped sample, these weights will be of the form k/n, for k = 0, 1, 2, . . .. When we cross-validate a bootstrap sample, we randomly divide the original observations into the two groups, but their bootstrap weights go with them (including the 0s).