Difference between re-training initial layers and final layers of NN

Question

I was going through a brief tutorial on "transfer learning" available here.

In this blog, a distinction has been made between training the initial layers and training the dense layers of a neural network. The author highlights that when the data similarity is less between two domains, it is better to fine-tune the lower layers of the neural network while it is more efficient to train the dense layers of the neural network when the data similarity is significant.

In this context, could anyone explain to me the logistics that governs the use of one strategy against the other?

Typically, I want to know in what way will the final model behave differently when the initial layers are pre-trained versus when the dense layers are pre-trained?

Answer 1

The way deep neural networks are usually understood is that the first layers do a "feature extraction", thus they extract and transform the input data into a form that is useful for the task at hand (e.g. classification or regression). The layers towards the end of the neural network than do the actual classification or regression. So I would assume that if your data is similar, it makes sense to use the same feature extraction, and you can leave the initial layers unchanged. If your data is however not very similar, you probably need a different type of feature-extraction, and you thus have to fine-tune the initial layers.

Answer 2

I think it is because pretraining is both helpful for generalization and optimization.

When the input data similarity is significant between two domains the generalization works here because the lower layers learn the basic features shared by the domains, for instance, we often use BERT for feature extraction for downstream NLP tasks.

While when the input data similarity is less the dense layers can supervise/guide the lower learnable layers to learn, functioning as optimization, for example, we can employ some high-level layers for a style transfer model. Take for one more example, we can use the last several layers in BERT for the ASR task since the input data for BERT is text but for ASR the input data is the sound wave but the result/target is the same: text. In this case, the parameters of the layers can be involved in the loss function, like the use of Gram matrix in style transfer learning.