Difference between "transfer learning" and "fine-tuning"

Question

I am currently reading the BERT paper and it splits the use of pre-trained models into two categories:

• Feature-based whereby you just take the embeddings for the tokens and plug it into whatever problem you want
• Fine-tuning where you add a couple of task-specific layers at the end of the pretrained model and then just run a couple of training epochs on that.

But in section 2.3 they mention transfer learning as if it were something different from "fine-tuning".

What is the difference (if any) between transfer learning and just fine-tuning as applied to LMs and downstream tasks?

Answer 1

Both terms are used as near-synonyms nowadays, but there is a slight difference.

• Transfer learning is a machine learning paradigm where you adapt a model trained for one task (in the case of BERT, it is masked language modeling) to perform another task (e.g., question answering)

• Finetuning means training a neural network (or only a sub-network) on a smaller dataset than the main training procedure, typically with a smaller learning rate and often using a different training objective. This is how transfer learning is done in the BERT paper.

There are instances of transfer learning without fine-tuning and vice versa. A popular way of transfer learning with Transformer is Adapters, which adds small subnetworks, but does not involve finetuning. Finetuning on artificially noised data is a common training step when training machine translation models (so they can handle typos). However, it is not transfer learning because the task is still the same.