Question about early stopping

Question

I've got a question regarding early stopping and I was hoping to get some insights here!

When would be best to stop training for a model that converges after a few epochs. It would make sense to stop training after training loss drops below the validation loss, but even after this point, validation loss keeps on decreasing, so would it be safe to assume the model is improving? or should the training be stopped when training loss drops below the validation loss?

I've attached two examples below, where in your opinion would be the best epoch to stop training? Also, as in the second example, the model converges quickly, does that mean the learning rate should be decreased even further (it's currently set as 0.0001, with an Adam optimizer).

Thanks!

Answer 1

To avoid overfitting on the training dataset, we use Early Stopping as a form of regularization. While using Early Stopping, there are three factors to consider,

First is what metric to track, here in your case, it is validation loss. Second factor is Patience; it is the number of epochs with no improvement after which training would be stopped. Third is to define what qualifies as an improvement, a minimum value of the absolute change from epoch to epoch in the tracked metric. There are no standard values for these, I'll link these answers for your reference.

In addition to that, which might also clear your second question too, I'd suggest you to try reducing the learning rate when your tracked metric has stopped improving (hitting a plateau) or decaying the learning rate by a fixed factor every few epochs. Both Tensorflow and PyTorch have libraries to do this for you.