2
$\begingroup$

I am trying to implement leave-one-out cross-validation from scratch.

I have a logistic regression model which I have already implemented. I have trained this model for 10,000 epochs. I am trying to update this to use LOOCV.

From what I understood, the LOOCV works by splitting the dataset into two sets:

  • one with n-1 examples in it. (training set)
  • and the other group with one example in it. And we use this one to test our model(validation set)

And we repeat this process n times.

Now, my questions are:

  1. For each of those n loops, we train our data. And like I said I did the initial training for 10,000. So are we going to train each of the n-models for 10,000 epochs? Does this mean we are looping over (10,000 * n) times during this whole process?
  2. Since we ran this process for n times. Does this mean we have n-different weights and bias? What weights and bias do I use for final testing? Do I use the best one from those n-models, or do I average over all the different weights and bias?

Thanks

$\endgroup$

2 Answers 2

2
$\begingroup$
  1. Right, but the number of epochs is your choice. You can decrease it a bit if you want LOOCV. Since you're going to train the model from scratch for each training fold, it'll be $e\times n$ looping over, where $e$ is your epoch number.

  2. Yes, you'll have n different set of parameters. Averaging is not good, even if the problem is convex like this one. For testing, you need to retrain using your training set.

$\endgroup$
5
  • $\begingroup$ so, just use Logistic regression using the training set without using LOOCV? $\endgroup$
    – ray an
    Feb 4, 2021 at 11:29
  • $\begingroup$ yes, cross validation (inc. loocv) generally has two uses: 1) hyperparameter opt 2) estimating the future test performance (esp. in small datasets where you don't have the luxury to separate a test set). What's your purpose of using CV? $\endgroup$
    – gunes
    Feb 4, 2021 at 11:34
  • $\begingroup$ This is one of the questions in my college assignment. We are told to use LOOCV to test the classifier. I do have a small dataset for this question. $\endgroup$
    – ray an
    Feb 4, 2021 at 11:58
  • $\begingroup$ It makes sense to use loocv. There is no separate test set, so you don't need to worry about the final model. $\endgroup$
    – gunes
    Feb 4, 2021 at 12:51
  • $\begingroup$ sure. Thank You $\endgroup$
    – ray an
    Feb 4, 2021 at 13:07
2
$\begingroup$

Question 1: Yes, you do train the model $n$ times, each time on an $n-1$-sized dataset. In some cases it is possible to save computational resources by using algebraic tricks to relate the estimates in each of the $n$ cases. You can do this e.g. in linear regression so that effectively your computational cost is that of a single fit (!); i.e. you are roughly $n$ times more effective than using the naive, brute force approach. But conceptually, you do fit the model $n$ times in LOOCV.

Question 2: Yes, you have $n$ different sets of weights and biases. You do not use any of them for out of sample purposes. Instead, you reestimate the model on the entire dataset and use the estimates from it.

$\endgroup$
2
  • $\begingroup$ when you say "you reestimate the model on the entire dataset and use the estimates from it" - does this mean just use the logistic regression model trained on the entire dataset without LOOCV $\endgroup$
    – ray an
    Feb 4, 2021 at 11:28
  • $\begingroup$ @rayan, yes. (LOOCV is typically used for hyperparameter tuning and/or for estimation of the model's out-of-sample performance.) $\endgroup$ Feb 4, 2021 at 11:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.