How to use early stopping properly for training deep neural network? I have a deep neural network model and I need to train it on my dataset which consists of about 100,000 examples, my validation data contains about 1000 examples. Because it takes time to train each example (around 0.5s for each example) and in order to avoid overfitting, I would like to apply early stopping to prevent unnecessary computation. But I am not sure how to properly train my neural network with early stopping, several things I do not quite understand now:


*

*What would be a good validation frequency? Should I check my model on the validation data at the end of each epoch? (My batch size is 1)

*Is it the case that the first few epochs might yield worse result before it starts converging to better value? In that case, should we train our network for several epochs before checking for early stopping?

*How to handle the case when the validation loss might go up and down? In that case, early stopping might prevent my model from learning further, right?
Thank you in advance.
 A: To add to other excellent answers, you can also - not stop. I usually:


*

*run NN for far more time I would have thought is sensible,

*save the model weights every N epochs, and

*when I see the training loss has stabilized, I just pick the model with lowest validation loss.


Of course that only makes sense when you don't pay by minute (or the cost is small enough) and when you can stop the training manually. The upside is that it is far easier to determine lowest validation error in hindsight.
A: 
What would be a good validation frequency? Should I check my model on the validation data at the end of each epoch? (My batch size is 1)

There is no gold rule, computing the validation error after each epoch is quite common. Since your validation set much smaller than your training set, it will not slow down the training much.

Is it the case that the first few epochs might yield worse result before it starts converging to better value? 

yes

In that case, should we train our network for several epochs before checking for early stopping?

You could, but then the issue is how many epochs should you skip. So in practice, most of the time people do not skip any epoch.

How to handle the case when the validation loss might go up and down? In that case, early stopping might prevent my model from learning further, right?

People typically define a patience, i.e. the number of epochs to wait before early stop if no progress on the validation set. The patience is often set somewhere between 10 and 100 (10 or 20 is more common), but it really depends on your dataset and network. 
Example with patience = 10:

