Fitting model on whole dataset, more or less epochs ? (w.r.t validation accuracy) [duplicate]

Question

When tuning my neural networks hyperparameters I use 20% of the data set as validation data. With the holdout set I observe the validation accuracy and validation loss. In my case the model starts overfitting after 150/300 epochs, so with early stopping I dump the model state at 150 epochs.

When fitting the final model where 100% of the data is used instead of 80%, should I assume the overfit might occur again at ~ 150 epochs, or could I continue training, or, stop earlier?

(The other hyperparameters have been tuned seperatly)

Answer 1

Let $T=\{\mathbf{X}_{train}, \mathbf{y}_{train}\}$ denote the training set and $V=\{\mathbf{X}_{val}, \mathbf{y}_{val}\}$ denote the validation set. Let also $i^*$ and $\boldsymbol{\theta}^*$ denote respectively the number of epochs and the parameters vector identified by the early stopping algorithm, training on $T$ and monitoring the loss on $V$. Now you have two choices:

• (most common) you can aggregate $T$ and $V$ together, obtaining a new training set $D$,

  $$ D=\left( \begin{bmatrix} \mathbf{X}_{train} \\ \mathbf{X}_{val} \end{bmatrix}, \begin{bmatrix} \mathbf{y}_{train} \\ \mathbf{y}_{val} \end{bmatrix} \right) $$

  reinitialize the weights (i.e., $\boldsymbol{\theta}=\boldsymbol{\theta}_0$ where $\boldsymbol{\theta}_0$ is a suitable random initialization for your model), and train for $i^*$ epochs. In other words, you consider $i^*$ as any other hyperparameter, which you tuned on the validation set: when you retrain on $D$ you don't change it, in the same way in which you wouldn't change the learning rate, that you tuned on the validation set, when you retrain on $D$. This choice (reinitialize weights and train for or $i^*$ epochs on $D$) has the advantage that the algorithm will always terminate. A disadvantage is that we assumed the number of epochs when training on $D$ should be the same found when training on $T$ with early stopping. But should the number of epochs be the same, or the number of parameter updates? Since $\vert T \vert <\vert D\vert$, an epoch on $D$ consists of more mini-batch updates (i.e., more parameter updates) than on $T$. We don't know which choice is better in general.

• (less common) $D$ is formed as in the other strategy. Now, however, rather than throwing $\boldsymbol{\theta}^*$ into the dustbin and restarting from scratch, we record the loss $\mathcal{L}_{val}^{final}$, the loss on $V$ at the end of the early stopping phase, and we continue training (on $D$), starting from $\boldsymbol{\theta}^*$. When do we stop? We stop when $\mathcal{L}_{val} < \mathcal{L}_{val}^{final}$ (note that we're training on $D$, but the stopping criterion monitors the loss on $V$). This choice has the advantage that we don't waste all the work done in training the model, but it has the disadvantage that the training phase is not as well behaved. As a matter of fact, we don't know if, after "growing" the training set, the loss on $V$ will ever go below the value it attained at the end of the early stopping phase. Thus, this procedure is not even guaranteed to terminate.