I have been using the Keras callback EarlyStopping to stop my model once the validation error has stopped decreasing. There's an option restore_best_weights in this callback which (if enabled) essentially resumes the weights from 'best epoch' (weights corresponding to lowest validation error) at end of each epoch. I am confused over the usage of this option. So far, I had been training my models without resuming the best weights but I think the loss landscape should be different for optimising the model with resuming the best weights at each epoch vs optimising from the current weights. Any idea which case should be preferred and why?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
5
One pattern of this early stopping arises when the validation loss decreases and then starts increasing. When the loss is increasing, this means parameters are moving away from the parameters that generalized the best.
The model's progress probably will be different when you compare restarting from the best epoch and starting again from the most recent epoch. When you restart training, and you have the goal of obtaining a lower loss than the best loss value during the previous training, you should re-start from the best loss value.
- SGD can give different results when the data have their order changed. This is what makes it stochastic.
- When re-starting from a location with higher validation loss, the model will parameters will have to move more to achieve a new best loss value, since the model loss has increased, and now will have to decrease again just to achieve its most recent minimum. Further decrease beyond that recent minimum will require the parameters to move even more.
-
$\begingroup$ I see. I will compare the behavior of two on my application. Theoretically restarting from the best weights is like 'trying to decrease loss with the same weights'. How about the usefulness of 'recent epoch weights' in making the model get out of bad optima? I mean if the model is re-using the same weights again and again, the probability to further decrease the loss might be low as compared to starting from 'new fresh point'? $\endgroup$ Commented Jun 9, 2020 at 13:26
-
1$\begingroup$ "Theoretically restarting from the best weights is like 'trying to decrease loss with the same weights'." It's not, because SGD is stochastic, so the optimizer will probably take a different path though the loss surface; this could result in a different or better model. "I mean if the model is re-using the same weights again and again," again, it's not reusing the same weights, it's continuing optimization from the same starting point. Starting from a new point sounds appealing, until you consider the challenges of picking that point. Initializing at random discards all of your progress. $\endgroup$– Sycorax ♦Commented Jun 9, 2020 at 13:43
-
$\begingroup$ Keras callback EarlyStopping does not stop training if restore_best_weights option is enabled. Despite of lowest validation error achieved and patience parameter enabled. Any idea what could be the issue? $\endgroup$ Commented Jun 10, 2020 at 6:07
-
$\begingroup$ Another thing I observed is, without restore_best_weights enabled, I get low validation error much faster than with restore_best_weights enabled (best-weights restored after end of each epoch). $\endgroup$ Commented Jun 10, 2020 at 6:09
-
$\begingroup$ Seems like this is a known phenomenon (but perhaps not a bug) github.com/keras-team/keras/issues/12511 $\endgroup$– Sycorax ♦Commented Jun 10, 2020 at 12:46