I am doing NLP, and I have this block of Transformer body that was already trained on dataset A. Now I am interested in fine tuning this same Transformer on a new dataset B.

In my Python code, should I re-initialize my optimizer and my scheduler before I try to fine tune my neural network on the different dataset?

Thank you,


Reinitializing the optimizer would probably reset momentum information. Either way, it would probably not make a massive difference.

Learning rate schedule is important -- fine-tuning is typically done with a much smaller learning rate, so as not to "mess up" the already carefully learned weights. You should probably use a completely different learning rate schedule for fine-tuning.

  • $\begingroup$ Hello, Thank you very much for your reply. So say I am using the linear schedular with warm up for both pre-training and fine-tuning. If so, would it be better to not initialize the linear schedular for fine tuning, since we want our learning rate to be small for fine-tuning? Thank you, $\endgroup$
    – HDB
    Nov 30 '19 at 0:30
  • $\begingroup$ yeah, that sounds right $\endgroup$
    – shimao
    Nov 30 '19 at 0:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.