I am reading Cross Validation with Confidence by Jing Lei. In section two he introduces cross-validation and makes a law of large numbers argument that CV gives you a good estimate.
Essentially, for a hold-out data point $(X, Y)$, we consider the loss $\ell(\hat f(X), Y)$ conditional on our fit model $\hat f$. The population equivalent is $\mathbb{E} \left[ \ell(\hat f(X), Y) \vert \hat f \right]$, which is what we would like to estimate.
Now, if we have multiple held-out data points, say $(X_{n+1}, Y_{n+1}), (X_{n+2}, Y_{n+2}), ...$, and we get corresponding estimates of out-of-sample loss $\ell(\hat f(X_{n+1}), Y_{n+1}), \ell(\hat f(X_{n+2}), Y_{n+2}), ...$, we can make an LLN argument that their average converges to our estimand provided $\ell(\hat f(X_{n+1}), Y_{n+1})$ and $\ell(\hat f(X_{n+2}), Y_{n+2})$, etc, are independent.
I see how $\epsilon \perp X$ is enough for independence of out of sample losses, but not how $\mathbb{E}( \epsilon | X) = 0$ (the condition in the text) is sufficient for this independence.
