My understanding is that self-supervised learning approaches approximately work like the following (I have Wav2Vec 2 in my mind here, used in speech recognition, but NLP transformer models are similar):
- You are training an "context representation" or "embedding" function $f_\theta$ minimizing (some loss involving) a contrastive loss on unlabelled data. This happens at a pre-training phase with data not from your domain, and $f_\theta$ a huge model you get from somewhere (Huggingface etc.).
- You fine-tune a classifier $g_\phi$, taking as input the context representation, on labelled data $D$, with $f_\theta$ frozen, minimizing a prediction loss (cross entropy etc.). $D$ is from your domain, but much smaller than in pre-training.
- Predictions are made using $g_\phi \circ f_\theta$.
Now suppose you want to do semi-supervised learning: your domain data $D$ consists of a labelled part, plus a lot of unlabelled observations. The next best thing would be to modify fine-tuning in (2) to use some off-the-shelf semi-supervised method (e.g., pseudo-labelling) and train $g_\phi$ utilizing all the labelled and unlabelled data.
My question: is it not also a sensible approach instead to follow both steps (1) and (2), and fine-tune $f_\theta$ using the unlabelled part of $D$, and then $g_\theta$ using the labelled part? Combining more contrastive learning and regular supervised learning instead of switching to a different method?
I have not been able to find any work which investigates this setting (it is hard to search for -- "semi-supervised" just leads to false positives talking about the self-supervision). Thinking about it, it feels like the more straigh-forward solution, instead of a new semi-supervised method; on the other hand, I would not be suprised if it was futile, as the new information provided to the contextual representation is very small to whatever was available in pre-training, leading to the unlabelled part contributing almost nothing.
