I have noticed that (from the original BERT paper) in the MLM training procedure, the authors decide to mask 15% of the words in a sentence. The mask works as following:

The masked words are distributed as follows

  1. 80% are replaced with [MASK] token (which makes perfect sense, just teach the model to learn some words given the left and right context)
  2. 10% are replaced by some random word. This makes some sense to me (https://stackoverflow.com/questions/64013808/why-bert-model-have-to-keep-10-mask-token-unchanged). My understanding is that this way the model learns to be influenced from the word it is trying to predict. That is, it does not consider only the left and right part of the sentence, but also the word itself. So masking with some random word would teach the model to actually consider the mirror words, but since the percentage is very small (1.5%), it would not confuse the model so much, so this might be beneficial.
  3. 10% of the words are unchanged. Now I completely don't understand this. For example, I don't understand the difference between: {90% masked with [MASK] and 10% masked with random word} and {80% [MASK], 10% random, 10% same word}. The authors indicate: The purpose of this is to bias the representation towards the actual observed word. Isn't this the exact purpose of the random mirror word placed? The only thing that makes sense to me is that random word replacement teaches the model to consider the mirror word, and the same-word counters the effect of the random word so that the model doesn't get confused logically?

1 Answer 1


The answer to your question is in §3.1 of the paper.

First, bear in mind that only the “masked” tokens (about 15%) are predicted during training, not all tokens. With that in mind, I would teach it in the reverse order of what’s in the paper. This ordering shows the value of predicting the observed word.

  1. It’s a normal, ordinary thing done for decades to predict the word in a position given its context. They do this, too, so that the representations of a word and its context are encouraged to be similar. BERT isn’t autoregressive, so it winds up seeing the word already (sort of like autoencoding), but the value is in relating the word to its context.
  2. For robustness, they also predict the right word when provided the wrong word: a randomly chosen one. This forces the model to lean on context more than on the word itself.
  3. Finally, further increasing the value of context, we train when no word at all is provided, learning to fill the slot only from the context.
  • $\begingroup$ Ooooh, wait a second: So when we have the sentence "The sun is nice" and we map "sun" to "hey" so we have "The hey is nice", then when the loss is computed, if the model correctly predicts "sun", there will be no penalty? My whole understanding was different, I thought that the model WILL get penalized for making the right prediction, this is what bothered me. But now I understand that putting the wrong word just makes the model also consider the context, not only the word itself. Also I understand your explanation of putting the same word: the enc. of the word shall be close to that of contex $\endgroup$
    – Petar Ulev
    May 12 at 15:33
  • $\begingroup$ Thanks so much. $\endgroup$
    – Petar Ulev
    May 12 at 15:33
  • $\begingroup$ It definitely deserves upvote and I will upvote it once I make 15 points. Currently I have 13 and can't upvote posts... But I will come back to it and upvote. Thanks again!! $\endgroup$
    – Petar Ulev
    May 12 at 21:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.