17
$\begingroup$

I have a deep neural network model and I need to train it on my dataset which consists of about 100,000 examples, my validation data contains about 1000 examples. Because it takes time to train each example (around 0.5s for each example) and in order to avoid overfitting, I would like to apply early stopping to prevent unnecessary computation. But I am not sure how to properly train my neural network with early stopping, several things I do not quite understand now:

  • What would be a good validation frequency? Should I check my model on the validation data at the end of each epoch? (My batch size is 1)

  • Is it the case that the first few epochs might yield worse result before it starts converging to better value? In that case, should we train our network for several epochs before checking for early stopping?

  • How to handle the case when the validation loss might go up and down? In that case, early stopping might prevent my model from learning further, right?

Thank you in advance.

$\endgroup$
  • $\begingroup$ I strongly recommend a batch size greater than one. Usual sizes are 32, 64, and 128. $\endgroup$ – ComputerScientist Dec 10 '18 at 19:37
17
$\begingroup$

What would be a good validation frequency? Should I check my model on the validation data at the end of each epoch? (My batch size is 1)

There is no gold rule, computing the validation error after each epoch is quite common. Since your validation set much smaller than your training set, it will not slow down the training much.

Is it the case that the first few epochs might yield worse result before it starts converging to better value?

yes

In that case, should we train our network for several epochs before checking for early stopping?

You could, but then the issue is how many epochs should you skip. So in practice, most of the time people do not skip any epoch.

How to handle the case when the validation loss might go up and down? In that case, early stopping might prevent my model from learning further, right?

People typically define a patience, i.e. the number of epochs to wait before early stop if no progress on the validation set. The patience is often set somewhere between 10 and 100 (10 or 20 is more common), but it really depends on your dataset and network.

Example with patience = 10:

enter image description here

$\endgroup$
  • $\begingroup$ Should one use the model when early stopped or the model patience epochs before stopped(the 'best' one with no further improvements) ? $\endgroup$ – Edityouprofile Mar 30 '17 at 7:46
  • 2
    $\begingroup$ @displayname patience epochs before stopped $\endgroup$ – Franck Dernoncourt Mar 30 '17 at 14:18
9
$\begingroup$

I recently came across a paper titled "Early Stopping -- but when?" by Lutz Prechelt that has many great examples of how to use early stopping with clear explanations of what each does and formulas for them.

Hopefully taking a look at that may help.

$\endgroup$
  • $\begingroup$ The paper is written in a weird style but the info was helpful. $\endgroup$ – Josiah Jul 24 '18 at 14:05
3
$\begingroup$

To add to other excellent answers, you can also - not stop. I usually:

  • run NN for far more time I would have thought is sensible,
  • save the model weights every N epochs, and
  • when I see the training loss has stabilized, I just pick the model with lowest validation loss.

Of course that only makes sense when you don't pay by minute (or the cost is small enough) and when you can stop the training manually. The upside is that it is far easier to determine lowest validation error in hindsight.

$\endgroup$
  • 1
    $\begingroup$ I do this too. Although like you I'm aware it's a luxury. When the models started taking up a lot of space, I found that using the previous loss and f-score figures I could programmatically determine when to delete previous models too - so you end up with a moving window of candidate best models that consume roughly as much hard disk space as they're likely to be worth. $\endgroup$ – QA Collective Jul 12 '18 at 8:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.