I have been using the BERT architecture implemented by the Huggingface library for my sentence classification task. Although, I read the paper (and related papers) and the result of my experiments is promising, However, I can not truly understand how the weights in the pre-trained model are actually helpful for the fine-tuned model.

I understand the intuition behind the pre-training and fine-tuning. My sentences are divided into tokens and the Masked Language Modeling (MLM) method tries to predict the masked tokens. The model learns some weights in the pre-training. What do these weights learn? I know the general idea is that the model learns a representation of the language. What are these representations? Are the weights learned for each token?

Suppose, the fine-tuning task is about sentence classification. To classify a sentence (using the same tokenizer, we will have the same tokens) how the pre-trained weights are helping?


1 Answer 1


What do these weights learn?

At the most literal level, they learn to predict the tokens of a sequence of text, given a corrupted form of that same sequence. In other words, they learn common patterns in language. BERT is a gigantic pseudo–language model.

Are weights learned for each token?

It’s easy for a misunderstanding to arise here. For the same word type occurring in different sentences (or in different places within the same sentence), the final representation will be different. Still, there are not parameters for each word token.

There are parameters for each word type, just like you’d find in a non-contextualized embedding (e.g. word2vec). The type embeddings of each token are passed through the BERT encoder network to produce vector representations of each token in context.

how [are] the pretrained weights helping?

TL;DR better initialization.

There are many local optima in the loss surface for large neural networks. Gradient-based optimization can converge to any of these, and which one you find depends on the initialization of the network. Some local optima are better than others. So how do we pick the right initialization, to give us a good local optimum? Especially with the vast number of parameters (and therefore, choices) in these models, we’ll take all the help we can get!

The intent of pretraining (and sequential transfer learning) is better initialization. If a parameter setting is better for task A (e.g. the BERT training objective), we suspect that it is close to a good parameter setting for task B (e.g. sentence classification). Start training on task B at the parameters you learned for task A.

Empirically, this approach seems to work well. The vast amount of training data for task A helps to get good performance on the related task B, which usually doesn’t have nearly as much data. If we just used the data for task B, we wouldn’t find as suitable of an optimum.

  • $\begingroup$ Thanks, @Arya McCarthy. Could you please explain (a bit more) the term "type" which you have used for word and embeddings in your answer? $\endgroup$
    – Adel
    Commented Apr 1, 2021 at 23:17
  • 1
    $\begingroup$ Sure! This comes from the type–token distinction in linguistics. The word type “hello” may occur numerous times in running text. Each is a token of the “hello” type. There is an embedding for the word type “hello” which is used for each token of the type. In case my explanation is fuzzy, here are Frank Ferraro’s slides explaining the difference. $\endgroup$ Commented Apr 1, 2021 at 23:22
  • $\begingroup$ Thank you @Arya McCarthy, the parameters that you mentioned are the "attention weights" or some weights for each type-token? $\endgroup$
    – Adel
    Commented Apr 2, 2021 at 1:19
  • 1
    $\begingroup$ There are tons of parameters (= weights) in the model. When I talked about parameters for each word type, I was referring to a matrix of weights. Each row is the embedding for one word type. $\endgroup$ Commented Apr 2, 2021 at 1:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.