# Isn't leave-1-out insufficient for proper classification evaluation?

I encountered several papers that used some classification method (for instance, LDA), with leave-1-out validation, and posted the classification results as an aggregation of all results (for all samples 'left out').

I was wondering if this weren't a little problematic. I'm used to putting, say, 20% of samples as test and then training the classifier on the remaining 80%. But in the leave-1-out, the case is more similar to 99% training and 1% testing (in the case of 100 samples, for the sake of example)... so how can I properly compare my method to leave-1-out?

Thanks.

## 1 Answer

The leave-one-out method is a cross-validation technique. If you have a data set with N=100 samples, you will run 100 train/test iterations where, in each iteration, the train set will have 99 test examples and 1 training example. This way you find out, for each sample, what the generated classification is for the sample when training on all other samples. You get a final error value by averaging the error over all iterations.

It is a particular case of k-fold cross validation (k=N) where you divide your data set into k equal-sized subsets and train on k-1 of them while you test on the remaining subset, and repeat that scenario for all combinations of k-1 training subsets and 1 test subset. You then get an error value by averaging the error of all iterations.

• In order to compare my performance with a 'leave-1-out' I would need to use a test set of a single sample? is it reasonable to assume (just to humor myself) that the performance of leave-1-out will be better than leave-k-out due to the lower number of training samples? – yoki Apr 18 '15 at 17:25
• In your example are you doing just a single iteration with 80% train and 20% test? To compare performance would be to compare the computed error value, and in the case of leave-k-out it would be an average of the error values from the iterations you did. I will edit my answer to reflect the averaging aspect of it. – adamconkey Apr 18 '15 at 17:39
• @yoki Yes, typically leave-one-out will have somewhat higher accuracies (or whatever) than, say, 10-fold cross-validation. To really compare methods you should use the same evaluation strategy. – Dougal Apr 18 '15 at 20:59