I was recently going through the Keras documentation on saving models. I am aware that saving a model involves saving the learned weights and biases after training. However, the doc also mentioned saving the optimizer state along with it.
Apparently, it allows resuming training from where you left off. But can someone explain what exactly does it save behind the scenes? And what does the optimizer have to do with the training state?
The optimizer state is the optimizer's momentum vector or similar history-tracking properties.
For example, the Adam optimizer tracks moving averages of the gradient and squared gradient. If you start training a model without restoring these data, the optimizer will operate differently. The updates will be different, so the optimizer will proceed along a different trajectory.
More details about adam: How does the Adam method of stochastic gradient descent work?
