Why BERT keep some masked tokens unchanged?

Question

As I understand, out of all masked tokens in BERT

1. Replace some with [mask], this is because of MLM
2. Replace some with other token, this will force model to generate proper contextual embedding for all tokens in the sequence, not only the [mask] ones. This is consistent with the goal of finetuning.

But I don't understand why BERT keep some masked tokens unchanged, could anyone please help me to understand it?

Answer 1

We can find the answer from the paper:

• 80% of the time: Replace the word with the [MASK] token, e.g., my dog is hairy → my dog is [MASK]
• 10% of the time: Replace the word with a random word, e.g., my dog is hairy → my dog is apple
• 10% of the time: Keep the word unchanged, e.g., my dog is hairy → my dog is hairy. The purpose of this is to bias the representation towards the actual observed word.

The purpose of BERT is to learn a representation for each token(conditioning on the rest of tokens and itself) and the representation relates not only to the rest of tokens but also the token itself, and one task becomes how to let our model learn when to depend on the embedding(a numerical feature of the token input into the model) and how it depends on it. Without the unchanged tokens, it just ignores the representation/embedding in the mirror position(note that the position embedding works and the embedding itself also can make it so) because the model learns that it would just be of no information(randomly chosen from the very large vocabulary or just MASK) in the training process and it just does the same in the inference mode(when we actually use it).

And we can continue reading:

The advantage of this procedure is that the Transformer encoder does not know which words it will be asked to predict or which have been replaced by random words, so it is forced to keep a distributional contextual representation of every input token. Additionally, because random replacement only occurs for 1.5% of all tokens (i.e., 10% of 15%), this does not seem to harm the model’s language understanding capability. In Section C.2, we evaluate the impact this procedure

Answer 2

I think we can interpret the problem in the following sense:

If we keep the masked tokens all masks that means the corresponding Y would be independent of the mask because the mask is the only choice and hence the independence is satisfied. No matter what the X is the Y remains the same distribution. And when we randomly choose the X that also satisfied.

The only way we can break the independence is we bias the representation towards the true words, thus the causation holds.

If the actual words are observed the distribution Y would be much different from if they are not observed(either all masks or random sampling).

Answer 3

The answer boils down to two points:

1. BERT's training loss, which guides its gradient descent, is not computed for all its input tokens, but only for the sampled ones.
2. The three tasks each have their own purpose:
   • Masking a sampled token (instead giving a generic, meaningless token) and predicting what it should be makes the model able to infer a token from only its context.
   • Replacing a sampled token by a random token and predicting what it should be makes the model robust against an erroneous token.
   • Leaving a sampled token unchanged and predicting it (in other words, auto-encoding the input) makes the model robust against erroneous contexts.

By point 1: if you didn't leave some sampled tokens unmasked, then even though most of the data flowing through BERT would be unmasked, the only embeddings that would be used in the loss would be anything that isn't the embedding of the token you want to predict (masked tokens start with the mask embedding, and random replacement tokens start with their own embeddings). That would impoverish the embeddings.