Why are language modeling pre-training objectives considered unsupervised?

Question

Maybe this is stemming from my not-so-great grasp of supervised vs. unsupervised learning, but my understanding is that if we have access to ground-truth labels then it's supervised learning and if not then it's unsupervised.

I'll take the masked language modeling (MLM) that BERT (Devlin et al., 2019) and many other subsequent language models use.

According to the original paper:

...we simply mask some percentage of the input tokens at random, and then predict those masked tokens... In this case, the final hidden vectors corresponding to the mask tokens are fed into an output softmax over the vocabulary, as in a standard LM.

If we just replace a certain percentage of tokens with [MASK] randomly, don't we technically have access to the ground-truth labels (i.e., the original unmasked tokens)? Shouldn't this be considered supervised learning?

My argument is analogous for the next sentence prediction (NSP) task.

---

Edit

I came across this blog post by Facebook AI Research (FAIR) regarding the concept of "self-supervised learning," and they also say that:

As a result of the supervisory signals that inform self-supervised learning, the term "self-supervised learning" is more accepted than the previously used term "unsupervised learning." Unsupervised learning is an ill-defined and misleading term that suggests that the learning uses no supervision at all. In fact, self-supervised learning is not unsupervised, as it uses far more feedback signals than standard supervised and reinforcement learning methods do.

The blog post talks about a lot of interesting topics, such as why SSL has impacted the field of NLP greatly but not so much for the field of CV, and the promise of energy-based models. Recommended reading.

https://ai.facebook.com/blog/self-supervised-learning-the-dark-matter-of-intelligence

Answer 1

Your argument is right. From the perspective of the language model, you have well-defined target labels and use supervise learning methods to teach the model to predict the labels.

Calling it unsupervised pre-training is certainly sort of paper-publishing marketing, but it is not entirely wrong. It is unsupervised from the perspective of the downstream tasks. The MLM-pre-trained model learned something useful for a particular downstream task (e.g., sentiment analysis) without using any labeled data for the task, but using unlabeled data only.

There is also a strong analogy with clustering. The inputs to the model are very high-dimensional data: the vocabulary has tens of thousand items, there is very little structure too (all one-hot vectors are equidistant). The MLM pre-training learns to embed such inputs into a much lower-dimensional and very structured space using nothing else than unlabeled data: the text itself.

Answer 2

The original GPT paper seems to consider the language modeling objective semi-supervised:

Our work broadly falls under the category of semi-supervised learning for natural language.

https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf