I encountered several papers that used some classification method (for instance, LDA), with leave-1-out validation, and posted the classification results as an aggregation of all results (for all samples 'left out').

I was wondering if this weren't a little problematic. I'm used to putting, say, 20% of samples as test and then training the classifier on the remaining 80%. But in the leave-1-out, the case is more similar to 99% training and 1% testing (in the case of 100 samples, for the sake of example)... so how can I properly compare my method to leave-1-out?



The leave-one-out method is a cross-validation technique. If you have a data set with N=100 samples, you will run 100 train/test iterations where, in each iteration, the train set will have 99 test examples and 1 training example. This way you find out, for each sample, what the generated classification is for the sample when training on all other samples. You get a final error value by averaging the error over all iterations.

It is a particular case of k-fold cross validation (k=N) where you divide your data set into k equal-sized subsets and train on k-1 of them while you test on the remaining subset, and repeat that scenario for all combinations of k-1 training subsets and 1 test subset. You then get an error value by averaging the error of all iterations.

  • $\begingroup$ In order to compare my performance with a 'leave-1-out' I would need to use a test set of a single sample? is it reasonable to assume (just to humor myself) that the performance of leave-1-out will be better than leave-k-out due to the lower number of training samples? $\endgroup$ – yoki Apr 18 '15 at 17:25
  • $\begingroup$ In your example are you doing just a single iteration with 80% train and 20% test? To compare performance would be to compare the computed error value, and in the case of leave-k-out it would be an average of the error values from the iterations you did. I will edit my answer to reflect the averaging aspect of it. $\endgroup$ – adamconkey Apr 18 '15 at 17:39
  • $\begingroup$ @yoki Yes, typically leave-one-out will have somewhat higher accuracies (or whatever) than, say, 10-fold cross-validation. To really compare methods you should use the same evaluation strategy. $\endgroup$ – Danica Apr 18 '15 at 20:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.