The answer to your question is in §3.1 of the paper.

First, bear in mind that only the “masked” tokens (about 15%) are predicted during training, not all tokens. With that in mind, I would teach it in the reverse order of what’s in the paper:

1. It’s a normal, ordinary thing done for decades to predict the word in a position given its context. They do this, too, so that the representations of a word and its context are encouraged to be similar. BERT isn’t autoregressive, so it winds up seeing the word already (sort of like autoencoding), but the value is in relating the word to its context.
2. For robustness, they also predict the right word when provided the wrong word: a randomly chosen one. This forces the model to lean on context more than on the word itself.
3. Finally, further increasing the value of context, we train when no word at all is provided, learning to fill the slot only from the context.

The answer to your question is in §3.1 of the paper.

First, bear in mind that only the “masked” tokens (about 15%) are predicted during training, not all tokens. With that in mind, I would teach it in the reverse order of what’s in the paper. This ordering shows the value of predicting the observed word.

1. It’s a normal, ordinary thing done for decades to predict the word in a position given its context. They do this, too, so that the representations of a word and its context are encouraged to be similar. BERT isn’t autoregressive, so it winds up seeing the word already (sort of like autoencoding), but the value is in relating the word to its context.
2. For robustness, they also predict the right word when provided the wrong word: a randomly chosen one. This forces the model to lean on context more than on the word itself.
3. Finally, further increasing the value of context, we train when no word at all is provided, learning to fill the slot only from the context.