BERT masking - why does it require sampling, and how does it mitigate the mismatch of the [MASK] token when fine-tuning?

Question

I'm reading the BERT paper and jalammar's illustrative guide for BERT.

I don't understand 2 things about the method's crux - the masked language model:

1. why does masking requires us to sample (take only 15%) of the words? can't we use the same sentence several times, each time masking another word? E.g. turn I am a student to I [mask] a student and I am a [mask]?
2. the authors were worried that the [mask] token itself would be used by BERT when training and then confuse it later at the fine-tuning stage (or even the inference stage). I don't understand the mitigation they use - in 10% of the times they replace [mask] with a random word, and in 10% they replace it back to the original word. How is that mitigating the problem? and if it does, why do they use such low percentages?

thanks, Ido

Answer 1

why does masking requires us to sample (take only 15%) of the words?

I think this guarantees that only about 1/7 of the words are masked, which is just like the window size in word2vec. That is on average in BERT we use 7 words as context to predict one word. The more words we mask the smaller the "window size" and the smaller the context.

Google did some experiments to try different corruption rates and here is the results: Source: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

I don't understand the mitigation they use - in 10% of the times they replace [mask] with a random word, and in 10% they replace it back to the original word.

This answer of mine would be of some help.