Stratification is usually defined as training data and testing data having the same distribution of class values. In leave-one-out cross-validation, each fold only has 1 instance. I know that it's impossible and unnecessary to apply stratification the way we do it for k-fold cross-validation. However, I don't know how to explain it. How do you explain that the single instance in the test data has the same distribution as the training data?


Maybe the difficulty is:

  • that in some cases stratification would be needed, but
  • due to leaving out exactly one instance at at time, it is impossible to achieve it.

In these situations, LOO is known to have a particularly large pessimistic bias. Textbook example is a majority vote classifer with two classes and the same number of cases in each class: the tested instance always comes from the minority class in the training data.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.