I read from Elements of Statistical Learning that the leave-one-out cross validation estimator has high variance, and I read the related stackexchange posts as to why this is the case 1. But I'm having trouble understanding why this is from a calculation stand point.

So the expected prediction error estimated by the LOOCV estimator is

$CV(\hat{f}) = \frac{1}{N}\sum_{i=1}^N (y_i-\hat{f}^{-\kappa(i)}(x_i)) =\sigma^2+bias(\hat{f})^2+var(\hat{f})$

where the model is $y_i=f(x_i)+\epsilon$, $\sigma^2$ is the irreducible error and the variance of $\epsilon$, and $\hat{f}^{-\kappa(i)}(x_i)$ is the LOOCV estimator trained on all samples except for the $i^{th}$ and evaluated at $x_i$.

I think the sample analogue for the bias and variance parts are (please correct me if I'm wrong!):

$bias(\hat{f})^2 = [\frac{1}{N}\sum_{i=1}^N (y_i-\frac{1}{N}\sum_{i=1}^N\hat{f}^{-\kappa(i)}(x_i))]^2$

$var(\hat{f})=\frac{1}{N}\sum_{i=1}^N[\hat{f}^{-\kappa(i)}(x_i)-\frac{1}{N}\sum_{i=1}^N\hat{f}^{-\kappa(i)}(x_i)]^2 $

Based on the sample analogue variance formula, it looks like the variance would be very low since $\hat{f}^{-\kappa(i)}(x_i)$ would be very similar across $i's$ due to significant overlapping between training sets, so the distance between each $\hat{f}^{-\kappa(i)}(x_i)$ and their average would be very low.

What's wrong with my reasoning here? Or is my sample analogue formula for MSE incorrect? If not, how can you tell from the sample analogue formula for variance that the variance is high?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.