# Why are language modeling pre-training objectives considered unsupervised?

Maybe this is stemming from my not-so-great grasp of supervised vs. unsupervised learning, but my understanding is that if we have access to ground-truth labels then it's supervised learning and if not then it's unsupervised.

I'll take the masked language modeling (MLM) that BERT (Devlin et al., 2019) and many other subsequent language models use.

According to the original paper:

...we simply mask some percentage of the input tokens at random, and then predict those masked tokens... In this case, the final hidden vectors corresponding to the mask tokens are fed into an output softmax over the vocabulary, as in a standard LM.

If we just replace a certain percentage of tokens with [MASK] randomly, don't we technically have access to the ground-truth labels (i.e., the original unmasked tokens)? Shouldn't this be considered supervised learning?

My argument is analogous for the next sentence prediction (NSP) task.

## Edit

I came across this blog post by Facebook AI Research (FAIR) regarding the concept of "self-supervised learning," and they also say that:

As a result of the supervisory signals that inform self-supervised learning, the term "self-supervised learning" is more accepted than the previously used term "unsupervised learning." Unsupervised learning is an ill-defined and misleading term that suggests that the learning uses no supervision at all. In fact, self-supervised learning is not unsupervised, as it uses far more feedback signals than standard supervised and reinforcement learning methods do.

The blog post talks about a lot of interesting topics, such as why SSL has impacted the field of NLP greatly but not so much for the field of CV, and the promise of energy-based models. Recommended reading.