0
$\begingroup$

I am currently reading the BERT paper and it splits the use of pre-trained models into two categories:

  • Feature-based whereby you just take the embeddings for the tokens and plug it into whatever problem you want
  • Fine-tuning where you add a couple of task-specific layers at the end of the pretrained model and then just run a couple of training epochs on that.

But in section 2.3 they mention transfer learning as if it were something different from "fine-tuning".

What is the difference (if any) between transfer learning and just fine-tuning as applied to LMs and downstream tasks?

$\endgroup$
2
  • 1
    $\begingroup$ Fine tuning is just one way of doing transfer learning. Indeed, they mention fine-tuning in Sec 2.3 (which is a related work section). $\endgroup$ Jan 23, 2023 at 3:39
  • $\begingroup$ I think the first approach is based on feature extraction, where you extract features from a pretrained model and use it as input of a (usually smaller) task-spefic network. In finetuning instead you add layers on top the pre-trained model and train the whole network with respect to a task-specific loss. $\endgroup$
    – Ciodar
    Jan 23, 2023 at 11:50

1 Answer 1

3
$\begingroup$

Both terms are used as near-synonyms nowadays, but there is a slight difference.

  • Transfer learning is a machine learning paradigm where you adapt a model trained for one task (in the case of BERT, it is masked language modeling) to perform another task (e.g., question answering)

  • Finetuning means training a neural network (or only a sub-network) on a smaller dataset than the main training procedure, typically with a smaller learning rate and often using a different training objective. This is how transfer learning is done in the BERT paper.

There are instances of transfer learning without fine-tuning and vice versa. A popular way of transfer learning with Transformer is Adapters, which adds small subnetworks, but does not involve finetuning. Finetuning on artificially noised data is a common training step when training machine translation models (so they can handle typos). However, it is not transfer learning because the task is still the same.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.