2
$\begingroup$

(Before the edit) Why do the authors of Introduction to Statistical Learning state that:

..the test error estimate resulting from LOOCV tends to have higher variance than does the test error estimate resulting from k-fold CV.

while LOOCV clearly yields the same results each time, thus resulting in no variance at all?

I know formulations of this question have already been answered and I read them all, but I just don't get what we're looking at.

EDIT: I looked again at the main points that are outlaid in the answers of the questions in the links Richard Hardy suggested

High variance of leave-one-out cross validation

The answer here says that we should be looking across different realizations of the dataset. I tried out to visualize this with an example, where I simulated a population of 1 000 000 observations from which I sampled a small training set on which I computed the k-fold cross validation error for K = 2, 5, 10 as well as LOOCV error. I repeated this 1000 times and recorded the errors for the different numbers of folds. The results were consistently showing a decrease in variance with an increase in the number of folds.

Happily in the post Bias and variance in leave-one-out vs K-fold cross validation the main answer suggests that this is not true in general, and might occur only in some special cases. The answer with just one upvote less, though, repeats the reasoning that's been given in the book ISLR. Correlated outcomes means higher variance, etc.

The same goes for the other posts on the subject. It seems people are contradicting about whether or not LOOCV has high variance or not.

I discovered though that no one is contradicting about the statement that there's correlation among the LOOCV estimates, and giving it a second thought I started doubting why this is. So my edited question goes:

Why are the test estimates correlated in LOOCV? I don't doubt that the trained models are correlated, but each of those correlated models is tested on an entirely different held out data point and therefore producing an entirely different test error which means that there isn't any correlation at all between the test estimates which we're averaging to obtain the LOOCV error.

For example: say there are 100 observations from the model Y = 2*X + eps, then all 100 models where one observation is held out will produce almost the same coefficient beta for the training model Y = (beta)*X. Testing on the held out observation the error is (Y-(beta)*X)^2, and if X has some variance var(X) than the test error's variance for the individual test observations will be a function of var(X). Therefore fully independent of the other test estimates and only dependent on the variability of X.

Does anyone have any thoughts on this that can make things clear for me?

Thanks in advance :)

$\endgroup$
  • $\begingroup$ There indeed are a few related threads with valuable answers: 1, 2, 3, 4, 5. Could you please identify how your question differs, thereby ensuring it is not a duplicate? Thank you. $\endgroup$ – Richard Hardy Aug 28 '19 at 19:46
  • $\begingroup$ Thank you for your response, I updated my question, please let me know if you still find it to be a duplicate. $\endgroup$ – DataFace Aug 30 '19 at 3:16
  • $\begingroup$ Looks clearer now, thank you! $\endgroup$ – Richard Hardy Aug 30 '19 at 5:53
0
$\begingroup$

Imagine you have you are testing your already build a model on one independent test set that has an outlier. This outlier has no way of influencing results on the other data points, thus here the test estimates are not correlated.

Now imagine the same situation but running CV instead. Now the same outlier does affect the results of other data points. You can say that this data-point is affecting your results twice, first, it's making your results worse when it's in the test set, second, it's making the results worse when it's in the training set because it's used to fit the models.

Of course, you don't need to have outliers in the data to observe this. This is just to illustrate how data points in CV are not completely independent as they would be in a one hold-out set scenario.

$\endgroup$
  • $\begingroup$ Thanks for your reply! I am not sure if I understand correctly. You're saying that testing on an independent test set gives uncorrelated estimates and testing using CV on the training set must have correlation? One other thing that also confuses me is how can one measure correlation between test estimates of a single method? $\endgroup$ – DataFace Aug 30 '19 at 16:30
  • $\begingroup$ Correlated is not the great word there, the better would be to think about it as being dependent. If you use some parametric test then it assumes independent samples (IID) which is not the case here $\endgroup$ – rep_ho Aug 30 '19 at 17:27
  • $\begingroup$ Great, thinking of the estimates as being dependent really works for me! Would this effect be bigger when using k-fold CV or LOOCV? For instance, in your example with an outlier? $\endgroup$ – DataFace Aug 31 '19 at 3:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.