5
$\begingroup$

Doing "leave one out cross validation" with a regression task is easy. You can calculate the MSE (mean squared error) even on one single sample and average them. But what about a classification task?

Calculating F1 Score, AUC, etc. on a single sample is not possible. So is "leave one out cross validation" possible on classification tasks?

You could just remember the decision of each single "leave one out cross validation" step. Build one confusion matrix after all cross validation steps are done and calculate the score from that. Is this how it is (can be) done?

2nd Question is: When I do "leave one out cross validation" I think doing early stopping is not possible (I cant stop on a single sample). Is there a solution to this dilemma?

Supplement: Early stopping is a method to avoid overfitting when training neural networks and gradient boosted trees (for example). You stop training when you see overfitting. Overfitting is measured on a validation set.

Improve this question
$\endgroup$
1
  • 2
    $\begingroup$ Note that mean squared error makes neither more nor less sense for a single regression case than AUC for a single classifier score: there's nothing to average, it's just a single squared error. But you can calculate that just as you can calculate a ROC curve and the area thereunder for a single predicted classifier score. (They don't hold that much information, but that's because they are based on a single data point only, plus the information loss due to the thresholding for the AUC) $\endgroup$
    – cbeleites unhappy with SX
    Feb 16, 2019 at 14:44

1 Answer 1

Reset to default
1
$\begingroup$

Your second approach is the right one. In LOOCV (just as in k-fold CV) you predict the class / probability for each observation and you save it. Then, at the end, once you have all the predictions for all the observations from all the folds, you build your confusion matrix on the whole data.

As for your second question, I do not understand. Why would you want to stop early?

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thanks for the answer. I added a supplement on my question to explain early stopping. $\endgroup$
    – Dieshe
    Feb 15, 2019 at 17:11
  • $\begingroup$ @Dieshe You are confusing the model building process and CV. When building sequential models you evaluate every k-th model (or every single model) with CV or a test set. These are two different processes, one builds a model, the second evaluates it. There is no reason to stop a CV, but you may be thinking of stopping the model building process. Some functions may allow this, check the documentation for such a feature. $\endgroup$
    – user2974951
    Feb 15, 2019 at 18:08
  • $\begingroup$ Yes - what I mean is stop the model building while doing CV. I do not mean to stop the CV. But I can not stop in a single Sample when doing "leave one out"... $\endgroup$
    – Dieshe
    Feb 15, 2019 at 19:16
  • $\begingroup$ Why can you not still stop early on a single sample? you still have access to the value of some loss function $\endgroup$
    – Tom
    Feb 15, 2019 at 19:44
  • $\begingroup$ @Tom This is purely an issue of implementation, if the function you are using allows this functionality then you can stop early. It has nothing to do with CV. $\endgroup$
    – user2974951
    Feb 16, 2019 at 9:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.