I'm looking into using Transfer Learning to take the ResNet50 model trained on ImageNet and fine tune it to my own dataset using Keras.

However, I feel I have some misconception about what exactly fine tuning is, and how to perform it.

In this paper, which I read many months back, I understood that transfer learning was a process where you took the first n layers from a pre-trained model, added on your own final layers for your task, and then fine tuning was where you did NOT freeze the weights from layers you transferred from the pre-trained model, but instead allowed them to update with a very low learning rate. I also understood that this method resulted in better generalisation, and results in general, than freezing the weights from the transferred layers.

However, every time I see fine tuning mentioned on the internet, people refer to freezing the weights from the transferred layers and applying a low learning rate to the new layers - only allowing their weights to update. As seen here.

This answer also recommends freezing the weights from the transferred layers.

I just don't see how this advice lines up with the results from the paper. It suggests that taking a large number of layers from the original network and freezing their weights will give a poor result, whereas allowing the weights from the transferred layers to be fine-tuned will improve generalisation.