1
$\begingroup$

During the last years I've been running cross validation procedures (and sometimes nested CV) to have an estimate of my model while also doing hyperparameter search.

The usual procedure that people mention online is that after having an estime of prediction power from a CV procedure, we just need to refit everything again on the whole dataset in order to have a final model.

However, how can one do that with a Neural Network? This is a problem for me because usually I don't select the model resulted from the last epoch's backpropagation. I select a model looking to some criteria in a validation set (for example, if I run a NN for 50 epochs, I select the model with the lowest loss in the validation set). Thus, how can we refit a neural network in the whole dataset if the choice of where to stop or which model to select throughout the epochs depend on a validation set to check/avoid overfitting? I mean, I'm considering that even if I define some number of epochs in the inner loop of a CV procedure, we don't know whether that will lead to some overfitting when training in the whole dataset...

Has anyone had this sort of issues before? How do you usually do a CV with a final model selection given we want it to be some sort of neural network?

$\endgroup$
1
$\begingroup$

I understand you are talking about how to use early stopping to fit a final model. Well, it basically works the same as during model selection: you keep a small portion of the data as validation to select at which epoch to stop. So, in this case, you can't use the whole dataset to fit the final model.

Notice that model selection is used to select not only hyper-parameters, but the whole fitting procedure (including preprocessing, early stopping and so on). Check out the answers in this topic: Nested cross validation for model selection, they are very enlightening.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.