2
$\begingroup$

When tuning my neural networks hyperparameters I use 20% of the data set as validation data. With the holdout set I observe the validation accuracy and validation loss. In my case the model starts overfitting after 150/300 epochs, so with early stopping I dump the model state at 150 epochs.

When fitting the final model where 100% of the data is used instead of 80%, should I assume the overfit might occur again at ~ 150 epochs, or could I continue training, or, stop earlier?

(The other hyperparameters have been tuned seperatly)

$\endgroup$
2
  • $\begingroup$ As discussed in the above linked thread, you cannot use this method as the number of actual weight updates per epoch (gradient descent steps) changes if you change the size of the training set. Also, the benefit of having a few more samples might not outweight the risk of overfitting, but this depends on the dataset size and task complexity... $\endgroup$ Jan 8, 2019 at 11:26
  • 1
    $\begingroup$ @JanKukacka it's not correct to say that you cannot use this method. It's more correct to say that in some cases it works, and in some others it doesn't, and you should be trying to keep the same number of parameter updates, rather than the same number of epochs. See deeplearningbook.org/contents/regularization.html, paragraph 7.8, and arxiv.org/pdf/1206.5533v2.pdf $\endgroup$
    – DeltaIV
    Jan 8, 2019 at 12:55

1 Answer 1

3
$\begingroup$

Let $T=\{\mathbf{X}_{train}, \mathbf{y}_{train}\}$ denote the training set and $V=\{\mathbf{X}_{val}, \mathbf{y}_{val}\}$ denote the validation set. Let also $i^*$ and $\boldsymbol{\theta}^*$ denote respectively the number of epochs and the parameters vector identified by the early stopping algorithm, training on $T$ and monitoring the loss on $V$. Now you have two choices:

  • (most common) you can aggregate $T$ and $V$ together, obtaining a new training set $D$,

    $$ D=\left( \begin{bmatrix} \mathbf{X}_{train} \\ \mathbf{X}_{val} \end{bmatrix}, \begin{bmatrix} \mathbf{y}_{train} \\ \mathbf{y}_{val} \end{bmatrix} \right) $$

    reinitialize the weights (i.e., $\boldsymbol{\theta}=\boldsymbol{\theta}_0$ where $\boldsymbol{\theta}_0$ is a suitable random initialization for your model), and train for $i^*$ epochs. In other words, you consider $i^*$ as any other hyperparameter, which you tuned on the validation set: when you retrain on $D$ you don't change it, in the same way in which you wouldn't change the learning rate, that you tuned on the validation set, when you retrain on $D$. This choice (reinitialize weights and train for or $i^*$ epochs on $D$) has the advantage that the algorithm will always terminate. A disadvantage is that we assumed the number of epochs when training on $D$ should be the same found when training on $T$ with early stopping. But should the number of epochs be the same, or the number of parameter updates? Since $\vert T \vert <\vert D\vert$, an epoch on $D$ consists of more mini-batch updates (i.e., more parameter updates) than on $T$. We don't know which choice is better in general.

  • (less common) $D$ is formed as in the other strategy. Now, however, rather than throwing $\boldsymbol{\theta}^*$ into the dustbin and restarting from scratch, we record the loss $\mathcal{L}_{val}^{final}$, the loss on $V$ at the end of the early stopping phase, and we continue training (on $D$), starting from $\boldsymbol{\theta}^*$. When do we stop? We stop when $\mathcal{L}_{val} < \mathcal{L}_{val}^{final}$ (note that we're training on $D$, but the stopping criterion monitors the loss on $V$). This choice has the advantage that we don't waste all the work done in training the model, but it has the disadvantage that the training phase is not as well behaved. As a matter of fact, we don't know if, after "growing" the training set, the loss on $V$ will ever go below the value it attained at the end of the early stopping phase. Thus, this procedure is not even guaranteed to terminate.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.