Why does LOOCV produce correlated estimates?

Question

(Before the edit) Why do the authors of Introduction to Statistical Learning state that:

..the test error estimate resulting from LOOCV tends to have higher variance than does the test error estimate resulting from k-fold CV.

while LOOCV clearly yields the same results each time, thus resulting in no variance at all?

I know formulations of this question have already been answered and I read them all, but I just don't get what we're looking at.

EDIT: I looked again at the main points that are outlaid in the answers of the questions in the links Richard Hardy suggested

High variance of leave-one-out cross validation

The answer here says that we should be looking across different realizations of the dataset. I tried out to visualize this with an example, where I simulated a population of 1 000 000 observations from which I sampled a small training set on which I computed the k-fold cross validation error for K = 2, 5, 10 as well as LOOCV error. I repeated this 1000 times and recorded the errors for the different numbers of folds. The results were consistently showing a decrease in variance with an increase in the number of folds.

Happily in the post Bias and variance in leave-one-out vs K-fold cross validation the main answer suggests that this is not true in general, and might occur only in some special cases. The answer with just one upvote less, though, repeats the reasoning that's been given in the book ISLR. Correlated outcomes means higher variance, etc.

The same goes for the other posts on the subject. It seems people are contradicting about whether or not LOOCV has high variance or not.

I discovered though that no one is contradicting about the statement that there's correlation among the LOOCV estimates, and giving it a second thought I started doubting why this is. So my edited question goes:

Why are the test estimates correlated in LOOCV? I don't doubt that the trained models are correlated, but each of those correlated models is tested on an entirely different held out data point and therefore producing an entirely different test error which means that there isn't any correlation at all between the test estimates which we're averaging to obtain the LOOCV error.

For example: say there are 100 observations from the model Y = 2*X + eps, then all 100 models where one observation is held out will produce almost the same coefficient beta for the training model Y = (beta)*X. Testing on the held out observation the error is (Y-(beta)*X)^2, and if X has some variance var(X) than the test error's variance for the individual test observations will be a function of var(X). Therefore fully independent of the other test estimates and only dependent on the variability of X.

Does anyone have any thoughts on this that can make things clear for me?

Thanks in advance :)

Answer 1

Imagine you have you are testing your already build a model on one independent test set that has an outlier. This outlier has no way of influencing results on the other data points, thus here the test estimates are not correlated.

Now imagine the same situation but running CV instead. Now the same outlier does affect the results of other data points. You can say that this data-point is affecting your results twice, first, it's making your results worse when it's in the test set, second, it's making the results worse when it's in the training set because it's used to fit the models.

Of course, you don't need to have outliers in the data to observe this. This is just to illustrate how data points in CV are not completely independent as they would be in a one hold-out set scenario.