i was fine-tuning this large language model with Stochastic Gradient Descent and mid epoch i stopped training, and saved the model weights. Then at a later time, reloaded the weights and restarted the model training. I see this dip and recovery in the validation accuracy. why is there a dip? I am using SGD so there shouldnt be any optimizer state that is reset.
$\begingroup$
$\endgroup$
Add a comment
|