I was reading about the bias-variance tradeoff associated with cross validation methods on James et al, Introduction to Statistical Learning (Page 183-184).
When we perform LOOCV, we are in effect averaging the outputs of n fitted models, each of which is trained on an almost identical set of observations; therefore, these outputs are highly (positively) correlated with each other.
What exactly is meant by "outputs of n fitted models"? For eg., if we are using linear regression, do these outputs refer to the model parameters?
Since the mean of many highly correlated quantities has higher variance than does the mean of many quantities that are not as highly correlated, the test error estimate resulting from LOOCV tends to have higher variance than does the test error estimate resulting from k-fold CV.
What exactly are the quantities being referred to in the above excerpt? For eg., if I have have 5 features in a data set of 100 observations and I want to use linear regression and LOOCV, I will get 100 different models where each model has 6 parameter estimates. Since the 100 models only differ by one observation, they are almost the same. When we talk about correlated quantities, are we talking about the parameters in these 100 models? And if yes, how does that lead to the test error estimate of LOOCV having a higher variance?