How pre-trained weights in the BERT can help the fine tuning task?

Question

I have been using the BERT architecture implemented by the Huggingface library for my sentence classification task. Although, I read the paper (and related papers) and the result of my experiments is promising, However, I can not truly understand how the weights in the pre-trained model are actually helpful for the fine-tuned model.

I understand the intuition behind the pre-training and fine-tuning. My sentences are divided into tokens and the Masked Language Modeling (MLM) method tries to predict the masked tokens. The model learns some weights in the pre-training. What do these weights learn? I know the general idea is that the model learns a representation of the language. What are these representations? Are the weights learned for each token?

Suppose, the fine-tuning task is about sentence classification. To classify a sentence (using the same tokenizer, we will have the same tokens) how the pre-trained weights are helping?

Answer 1

What do these weights learn?

At the most literal level, they learn to predict the tokens of a sequence of text, given a corrupted form of that same sequence. In other words, they learn common patterns in language. BERT is a gigantic pseudo–language model.

Are weights learned for each token?

It’s easy for a misunderstanding to arise here. For the same word type occurring in different sentences (or in different places within the same sentence), the final representation will be different. Still, there are not parameters for each word token.

There are parameters for each word type, just like you’d find in a non-contextualized embedding (e.g. word2vec). The type embeddings of each token are passed through the BERT encoder network to produce vector representations of each token in context.

how [are] the pretrained weights helping?

TL;DR better initialization.

There are many local optima in the loss surface for large neural networks. Gradient-based optimization can converge to any of these, and which one you find depends on the initialization of the network. Some local optima are better than others. So how do we pick the right initialization, to give us a good local optimum? Especially with the vast number of parameters (and therefore, choices) in these models, we’ll take all the help we can get!

The intent of pretraining (and sequential transfer learning) is better initialization. If a parameter setting is better for task A (e.g. the BERT training objective), we suspect that it is close to a good parameter setting for task B (e.g. sentence classification). Start training on task B at the parameters you learned for task A.

Empirically, this approach seems to work well. The vast amount of training data for task A helps to get good performance on the related task B, which usually doesn’t have nearly as much data. If we just used the data for task B, we wouldn’t find as suitable of an optimum.