32
$\begingroup$

I have a deep neural network model and I need to train it on my dataset which consists of about 100,000 examples, my validation data contains about 1000 examples. Because it takes time to train each example (around 0.5s for each example) and in order to avoid overfitting, I would like to apply early stopping to prevent unnecessary computation. But I am not sure how to properly train my neural network with early stopping, several things I do not quite understand now:

  • What would be a good validation frequency? Should I check my model on the validation data at the end of each epoch? (My batch size is 1)

  • Is it the case that the first few epochs might yield worse result before it starts converging to better value? In that case, should we train our network for several epochs before checking for early stopping?

  • How to handle the case when the validation loss might go up and down? In that case, early stopping might prevent my model from learning further, right?

Thank you in advance.

Improve this question
$\endgroup$
3
  • $\begingroup$ I recently came across a paper titled "Early Stopping -- but when?" by Lutz Prechelt that has many great examples of how to use early stopping with clear explanations of what each does and formulas for them. Hopefully taking a look at that may help. $\endgroup$
    – Pro Q
    Commented May 4, 2017 at 3:11
  • 1
    $\begingroup$ I strongly recommend a batch size greater than one. Usual sizes are 32, 64, and 128. $\endgroup$
    – ComputerScientist
    Commented Dec 10, 2018 at 19:37
  • $\begingroup$ Batch size is recommended to be about ~1-2 x the number of categories you are aiming to learn. training on ImageNet this amounts to somewhere between 1000-2000 samples per batch. $\endgroup$
    – bonobo
    Commented Sep 7, 2020 at 7:57

2 Answers 2

Reset to default
23
$\begingroup$

What would be a good validation frequency? Should I check my model on the validation data at the end of each epoch? (My batch size is 1)

There is no gold rule, computing the validation error after each epoch is quite common. Since your validation set much smaller than your training set, it will not slow down the training much.

Is it the case that the first few epochs might yield worse result before it starts converging to better value?

yes

In that case, should we train our network for several epochs before checking for early stopping?

You could, but then the issue is how many epochs should you skip. So in practice, most of the time people do not skip any epoch.

How to handle the case when the validation loss might go up and down? In that case, early stopping might prevent my model from learning further, right?

People typically define a patience, i.e. the number of epochs to wait before early stop if no progress on the validation set. The patience is often set somewhere between 10 and 100 (10 or 20 is more common), but it really depends on your dataset and network.

Example with patience = 10:

enter image description here

Improve this answer
$\endgroup$
4
  • $\begingroup$ Should one use the model when early stopped or the model patience epochs before stopped(the 'best' one with no further improvements) ? $\endgroup$
    – Saddle Point
    Commented Mar 30, 2017 at 7:46
  • 2
    $\begingroup$ @displayname patience epochs before stopped $\endgroup$
    – Franck Dernoncourt
    Commented Mar 30, 2017 at 14:18
  • $\begingroup$ Patience bakes all of that stuff in. When the model stops, it can be parameterized to load the best weights that it has found. Just put an excess of epochs 200? 45000? and let patience stop the process for you. $\endgroup$
    – Mike_K
    Commented Jan 3, 2021 at 22:45
  • $\begingroup$ As to skipping validation in early epochs, you may want to do this if you know from similar training runs that the results are of no interest, e.g. in the example in the answer one may want to skip validation in the first 5 epochs. $\endgroup$
    – Joachim Wagner
    Commented Apr 24 at 12:08
4
$\begingroup$

To add to other excellent answers, you can also - not stop. I usually:

  • run NN for far more time I would have thought is sensible,
  • save the model weights every N epochs, and
  • when I see the training loss has stabilized, I just pick the model with lowest validation loss.

Of course that only makes sense when you don't pay by minute (or the cost is small enough) and when you can stop the training manually. The upside is that it is far easier to determine lowest validation error in hindsight.

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ I do this too. Although like you I'm aware it's a luxury. When the models started taking up a lot of space, I found that using the previous loss and f-score figures I could programmatically determine when to delete previous models too - so you end up with a moving window of candidate best models that consume roughly as much hard disk space as they're likely to be worth. $\endgroup$
    – QA Collective
    Commented Jul 12, 2018 at 8:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.