1
$\begingroup$

Assume that you found the optimal hyperparameters of a neural network (e.g. a multi layer feed forward NN) with k-fold cross validation in a grid search. Lets assume you have varied the number of epochs and got the optimal epoch number as a result.

Now you want to train the network for the optimal number of epochs on the whole data used for CV. In my opinion following problem arises: When using a constant mini batch size, number of iterations varies for datasets of different size - so your optimal epoch number found for the smaller training split in the CV can not be applied to training with the whole CV data. Am I missing something here? Right at the moment I can not find a proper solution for finding the optimal training epochs for my NN. I searched the net for some time now, but could not find an answer.

P.S. I only described the problem after the inner loop of nested CV, but the same problem arises after the outer loop (training the model on ALL data, not only on CV data for putting the model to production).

$\endgroup$
2
$\begingroup$

The "optimal" number of epochs for a neural network training is not very reliable (i.e. has high variance), so generally I would recommend against it. The benefit you get from using a few more samples is usually less than the benefit a validation set gives you (early stopping). See this answer for more details.

Otherwise, you are correct. If you change the size of the dataset while keeping the batch size equal, an "epoch" corresponds to a different number of weight updates, which are the real measure of training length. You would have to work with the update count rather than the epoch count. See this answer for details.

$\endgroup$
  • $\begingroup$ After successfully building my NN with suggestions in the links you posted, I read into the topic k-fold cross validation again. Many posts on the internet recommend optimizing epoch numbers on NN with k-fold CV and different sizes of dataset. It seems many people get confused with that technique. As I see it a possible solution would be to stop training of the net if the train accuracy in each CV loop stops increasing significantly. Overfit would have to be handled solely by parameter optimization then. This is only an idea though and computational expensive. $\endgroup$ – Hoizmichl Mar 26 '18 at 7:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.