My understanding is that self-supervised learning approaches approximately work like the following (I have Wav2Vec 2 in my mind here, used in speech recognition, but NLP transformer models are similar):

  1. You are training an "context representation" or "embedding" function $f_\theta$ minimizing (some loss involving) a contrastive loss on unlabelled data. This happens at a pre-training phase with data not from your domain, and $f_\theta$ a huge model you get from somewhere (Huggingface etc.).
  2. You fine-tune a classifier $g_\phi$, taking as input the context representation, on labelled data $D$, with $f_\theta$ frozen, minimizing a prediction loss (cross entropy etc.). $D$ is from your domain, but much smaller than in pre-training.
  3. Predictions are made using $g_\phi \circ f_\theta$.

Now suppose you want to do semi-supervised learning: your domain data $D$ consists of a labelled part, plus a lot of unlabelled observations. The next best thing would be to modify fine-tuning in (2) to use some off-the-shelf semi-supervised method (e.g., pseudo-labelling) and train $g_\phi$ utilizing all the labelled and unlabelled data.

My question: is it not also a sensible approach instead to follow both steps (1) and (2), and fine-tune $f_\theta$ using the unlabelled part of $D$, and then $g_\theta$ using the labelled part? Combining more contrastive learning and regular supervised learning instead of switching to a different method?

I have not been able to find any work which investigates this setting (it is hard to search for -- "semi-supervised" just leads to false positives talking about the self-supervision). Thinking about it, it feels like the more straigh-forward solution, instead of a new semi-supervised method; on the other hand, I would not be suprised if it was futile, as the new information provided to the contextual representation is very small to whatever was available in pre-training, leading to the unlabelled part contributing almost nothing.


1 Answer 1


Yes, it is indeed a sensible approach and has been done before in e.g., ULMFiT and DAPT. Substantial improvements are observed in both cases.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.