Why does the masked language modelling (MLM) task produce useful embeddings?

Question

Masked language modelling is the standard way of training a language model such as a transformer. Each input token has some probability (e.g. 15%) of being replaced with a <MASK> token. The model must predict the original input token at these positions.

There are various variants of this task, such as sometimes replacing the token with a random token or the original token instead of <MASK>, but the original BERT paper (in the ablation studies in the appendix) found that the simple approach of always replacing with <MASK> works well.

My question is: why does this task produce useful embeddings for non-<MASK> tokens? This is the main use of such models, when we use their embeddings for downstream tasks. When the token is replaced with <MASK>, it makes sense to me that the model learns how to use the surrounding tokens to create a contextual representation of what the token is likely to be. But when the token is not <MASK>, the model should know that this is the correct token, and might as well just leave the token alone, yielding a perfect "prediction". In fact, for non-<MASK> tokens, there was no loss during training, so the model had no incentive to do anything in particular for these tokens. So why does it produce good embeddings?

Answer 1

You have answered your own question here (note my emphasis):

When the token is replaced with <MASK>, it makes sense to me that the model learns how to use the surrounding tokens to create a contextual representation of what the token is likely to be.

In other words, the contextual representation of a masked token is a function not only of its embedding but also the embeddings of the other tokens in the input, masked or otherwise. Thus, although the loss is only computed for the masked tokens, its gradient flows to all the token embeddings, i.e. they're updated during training. So, the model is "incentivized" to also produce good embeddings for the unmasked tokens.