I have read that leave-one-out cross-validation provides a relatively “unbiased estimate of the true generalization performance” (e.g. here) and that this is an advantageous property of the leave-one-out CV.
However, I don't see how this follows from the properties of leave-one-out CV. Why is the bias of this estimator low when compared to others?
I keep investigating the topic, and I believe it has to do with the fact that this estimator is less pessimistic than, say, K-fold validation, since it uses all the data but one instance, but it would be great to read a mathematical derivation of this.