I am reading Cross Validation with Confidence by Jing Lei. In section two he introduces cross-validation and makes a law of large numbers argument that CV gives you a good estimate.

enter image description here

enter image description here

Essentially, for a hold-out data point $(X, Y)$, we consider the loss $\ell(\hat f(X), Y)$ conditional on our fit model $\hat f$. The population equivalent is $\mathbb{E} \left[ \ell(\hat f(X), Y) \vert \hat f \right]$, which is what we would like to estimate.

Now, if we have multiple held-out data points, say $(X_{n+1}, Y_{n+1}), (X_{n+2}, Y_{n+2}), ...$, and we get corresponding estimates of out-of-sample loss $\ell(\hat f(X_{n+1}), Y_{n+1}), \ell(\hat f(X_{n+2}), Y_{n+2}), ...$, we can make an LLN argument that their average converges to our estimand provided $\ell(\hat f(X_{n+1}), Y_{n+1})$ and $\ell(\hat f(X_{n+2}), Y_{n+2})$, etc, are independent.

I see how $\epsilon \perp X$ is enough for independence of out of sample losses, but not how $\mathbb{E}( \epsilon | X) = 0$ (the condition in the text) is sufficient for this independence.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.