Confusion about the training procedure while using transfer learning

Question

Suppose that we have a trained CNN, there is 5 conv layers and 3 fully connected layers. We take the first 5 conv layers as it is (with their parameter settings: like kernel size, activation etc) and their weights and biases which are trained before by using another dataset.

If we want to profit from this knowledge in our new model (which has the same 5 conv layers at the beginning but later is different), do we:

• Continue to train the whole model (at the beginning: 5 conv layers is initialized to the parameters found in the previous training and the later layers some other initialization (e.g. he initialization for weights and 0 for biases)) with our data?

or

• We keep the first 5 conv layers as in the test mode (no more update) and only update/train the parameters in later layers?

Which one is understood/done when we talk about transfer learning?

Note: I am not very familiar with all the terminology used in deep learning. However, I hope that I could at least explain my question.

Answer 1

Stanford University lists three types of transfer learning.

1. Take a pretrained ConvNet on ImageNet, remove the last fully-connected layer (this layer’s outputs are the class scores for a different task), then treat the rest of the ConvNet as a fixed feature extractor for the new dataset.
2. Replace and retrain the classifier on top of the ConvNet on the new dataset, and also fine-tune the weights of the pretrained network by continuing the backpropagation. You can fine-tune all the layers of the ConvNet or keep some of the earlier layers fixed (due to overfitting concerns) and only fine-tune some higher-level portion of the network.
3. Fine-tune a published pre-trained model.

Note that the last type really only differs in that the pre-trained model is typically large and publicly available.

When to use which approach depends on how much data you have and how different it is from the pre-trained model (for more details see the link).

\begin{array} {|l|l|l|} \hline \bf \text{Amount of data}& \bf \text{Similarity of data}& \bf \text{Procedure} \\ \hline Low & Low & \text{Unfreeze more layers of pretrained model}\\ \hline Low & High & \text{Unfreeze fewer layers of pretrained model}\\ \hline High & Low & \text{Initialize with pretrained weights and train completely} \\ \hline High & High & \text{Fine-tune completely} \\ \hline \end{array}

CS231n Convolutional Neural Networks for Visual Recognition

Answer 2

Both approaches are called "transfer learning". Actually, it is quite common to combine the two. Then, the procedure looks like this:

1. Freeze all but the last few (often just one) layer; find good learning rate; train the unfreezed layers a couple of epochs.
2. Unfreeze a few additional layers (or all); find good learning rate; train the unfreezed part a couple of epochs.
3. Repeat

Answer 3

Generally transfer learning is used as an umbrella term for any procedure that could leverage pre-trained models for different tasks. Currently, it is de-facto practice in NLP and multi task learning and also Google's new architecture pathways.

In strict sense, transfer learning implies using weights coming from the previous experience, given architecture is fixed. Simple weight transfer might not be sufficient in this strict setting, all the state of the previous model should be retained. Then, training is resumed with the new data.

However, freezing earlier layers may not be full transfer-learning, as in changing the task but fine tuning for a closely related learning. It appears to be the "degree of how much to transfer" is controlled by freezing part of the network weights.

An other justification for freezing certain parts of the weights are to prevent catastrophic forgetting, but this appears to be again an open research field, see here

See also:

• Foundation models : Is it a new paradigm for statistics and machine learning?
• Transfer Learning on Autoencoders?
• How to resume training in neural networks properly?.