Implementation of Leave-one-out CV

Question

I am trying to implement leave-one-out cross-validation from scratch.

I have a logistic regression model which I have already implemented. I have trained this model for 10,000 epochs. I am trying to update this to use LOOCV.

From what I understood, the LOOCV works by splitting the dataset into two sets:

• one with n-1 examples in it. (training set)
• and the other group with one example in it. And we use this one to test our model(validation set)

And we repeat this process n times.

Now, my questions are:

1. For each of those n loops, we train our data. And like I said I did the initial training for 10,000. So are we going to train each of the n-models for 10,000 epochs? Does this mean we are looping over (10,000 * n) times during this whole process?
2. Since we ran this process for n times. Does this mean we have n-different weights and bias? What weights and bias do I use for final testing? Do I use the best one from those n-models, or do I average over all the different weights and bias?

Thanks

Answer 1

1. Right, but the number of epochs is your choice. You can decrease it a bit if you want LOOCV. Since you're going to train the model from scratch for each training fold, it'll be $e\times n$ looping over, where $e$ is your epoch number.

2. Yes, you'll have n different set of parameters. Averaging is not good, even if the problem is convex like this one. For testing, you need to retrain using your training set.

Answer 2

Question 1: Yes, you do train the model $n$ times, each time on an $n-1$-sized dataset. In some cases it is possible to save computational resources by using algebraic tricks to relate the estimates in each of the $n$ cases. You can do this e.g. in linear regression so that effectively your computational cost is that of a single fit (!); i.e. you are roughly $n$ times more effective than using the naive, brute force approach. But conceptually, you do fit the model $n$ times in LOOCV.

Question 2: Yes, you have $n$ different sets of weights and biases. You do not use any of them for out of sample purposes. Instead, you reestimate the model on the entire dataset and use the estimates from it.