2
$\begingroup$

I have trained a CNN with EarlyStopping and I wonder if I should not use EarlyStopping and waste 20% of Trainingsdata for Validation, because it looks like as that the validation loss doesn't increase after 50 Trainingsepochs (please see the image).

Sorry for this simple questions but I'm a beginner and I try to understand when EarlyStopping is really necessary and when it is superfluos to use EarlyStopping.

enter image description here

$\endgroup$
8
  • $\begingroup$ How are you splitting your data at the moment? 80% training and 20% test? $\endgroup$
    – Janosch
    Jan 14 '20 at 9:17
  • $\begingroup$ @Tim 20% Test, 20% Validation, 60% Training. $\endgroup$
    – Code Now
    Jan 14 '20 at 9:56
  • $\begingroup$ @Tim The problem is, that the training data set is very small, only 500 images. $\endgroup$
    – Code Now
    Jan 14 '20 at 10:42
  • $\begingroup$ A more common split is 80%|10%|10%. So you can retain the same amount training samples, but you validate on less $\endgroup$
    – Janosch
    Jan 14 '20 at 11:44
  • $\begingroup$ @Tim If I understand that correctly, you find according to the learning curve shown above that EarlyStopping should still be used? But only with 10% for validation. $\endgroup$
    – Code Now
    Jan 14 '20 at 11:52
4
$\begingroup$

One common way of splitting the data is into the 80%,10%,10%. EarlyStopping is used to prevent the model from overfitting. You could also do the "EarlyStopping" by hand.

You could run the model see at what point you overfit and then choose the model from the appropriate epoch (for which you need to save those models while training). The usage of EarlyStopping just automates this process and you have additional parameters such as "patience" with which you can adapt the earlystopping rules.

In your example you train your model for too long. You should definitely stop training the latest at epoch 30 where after the validation loss start to increase again. BUt you could already stop at epoch 10 as your loss only improves really slowly.

EarlyStoppping rules just help to automate this detection. But in general you should always stop training when the validation error increases.

It can be helpful to not only split 80:20 (but 80:10:10) because deciding to stop training based on the validation set can also overfit to the validationset.

$\endgroup$
8
  • $\begingroup$ Ok, the setting of patience should also depend on the learning rate? When I have a very low learning rate e.g. optimizer Adam with lr=0.0001, then would I have to assign a higher value to patience (e.g. 20 epochs or higher)? Or better, if I have enough time, and if I have set 'restore_best_weights=True' (Keras EarlyStopping) then I could also choose a higher value for patience? $\endgroup$
    – Code Now
    Jan 14 '20 at 12:13
  • $\begingroup$ While it makes sense to increase the patience when the learning rate is increased, they are not related and you should try to find a suitable lr independently of the patience you are using. To be honest I do not use keras that much so I am not sure how "restore_best_weights" is used. but I do know there is another module which you can add to your callbakcs which stores the best model $\endgroup$
    – Janosch
    Jan 14 '20 at 12:24
  • $\begingroup$ Maybe you allow me one more question. If you would like to optimize hyperparameters such as learning rate and dropout rate via grid search, would you then use a fixed number of epochs? $\endgroup$
    – Code Now
    Jan 14 '20 at 15:15
  • $\begingroup$ I am not sure what the consensus is here. But I would not (unless you care about speed and you want to train a model in a limited number of epochs), I would use the EarlyStopping that way you can guarantee that all models with different hyperparameters train to the "optimal" loss. In case of hyperparameter tuning you would need a validation and a test set. For example 80:10:10 $\endgroup$
    – Janosch
    Jan 16 '20 at 14:22
  • $\begingroup$ Y. Bengio recommend in his paper "Practical REcommendations for Gradient-Based Training of Deep Architectures" arxiv.org/abs/1206.5533 (see page 9-10) that it is useful to turn early-stopping off when analyzing the effect of individual hyper-parameters, because early stopping can hide overfitting effect of other hyperparameters. Therefore I wonder if this couldn't also happen via grid search when using early stopping for each model. $\endgroup$
    – Code Now
    Jan 16 '20 at 15:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.