5
$\begingroup$

Suppose we have a convolutional neural network trained for task A, and we wish to adapt it for a similar task B. Generally speaking, we preserve the convolutional weights and fully connected layers, and then fine-tune the network for the new task. Further simplifications include freezing the first portion of convolutional layers, and only training the last few convolutional layers.

The typical suggestion here is to use a reduced learning rate. However, this seems rather artificial to me. Specifically, if the CNN is for object classification, we strip the softmax layer and add a completely new softmax layer. So I would think that one should use a higher learning for the softmax layer, and possibly the fully connected layer just prior. I haven't really seen examples of this in practice, and I was wondering if it would make significant impact on the overall training speed.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
3
$\begingroup$

I don't know where you read about the reduced learning rate, but I think there was some misunderstanding.

The advice is to use a smaller learning rate for the weights that are being fine-tuned and a higher one for the randomly initialized weights (e.g. the ones in the softmax classifier). Pretrained weights are already good, they need to be fine-tuned, not distorted.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ If you look at finetuning examples in Keras and Tensorflow (Object detection), none of them heed this advice for retraining on new tasks. The closest I've seen is to freeze the first set of layers, train the model for one epoch, and then unfreeze all layers, and resume training with a lower learning rate. In this second epoch, the learning rate is the same for all unfrozen layers. $\endgroup$
    – Alex R.
    May 30, 2018 at 17:21
  • $\begingroup$ I guess that makes sense too: from a random state go quickly to a good state (high LR), then fine-tune everything with lower LR. I'd bet setting different learning rates for layers gets a better accuracy in less time though. That is also the advice given in Stanford cs231: cs231n.github.io/transfer-learning As usual, only experiments will tell what's the best approach $\endgroup$
    – iacolippo
    May 31, 2018 at 9:05
  • $\begingroup$ hey @iacolippo , what about over-fitting? pre-trained models are sensitive to overfit. & smaller learning-rate may increase chances of overfit $\endgroup$
    – Thunder
    Jun 15, 2021 at 2:53
0
$\begingroup$

Ideally, you can freeze the pre-trained weights and only train the softmax layer. Next, you can fine-tune the entire network with a reduced learning rate

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.