Stratification is usually defined as training data and testing data having the same distribution of class values. In leave-one-out cross-validation, each fold only has 1 instance. I know that it's impossible and unnecessary to apply stratification the way we do it for k-fold cross-validation. However, I don't know how to explain it. How do you explain that the single instance in the test data has the same distribution as the training data?
Maybe the difficulty is:
- that in some cases stratification would be needed, but
- due to leaving out exactly one instance at at time, it is impossible to achieve it.
In these situations, LOO is known to have a particularly large pessimistic bias. Textbook example is a majority vote classifer with two classes and the same number of cases in each class: the tested instance always comes from the minority class in the training data.