TL;DR:
The second model can add value by leveraging the unlabelled data to improve generalisation and potentially achieve better performance than the first model alone. However, this requires careful handling of pseudo-labels to avoid propagating errors. While it might seem that the second model is merely reinforcing the first model's predictions, the additional data and iterative improvement can lead to a model that generalises better to new, unseen data.
Your approach is a common strategy in semi-supervised learning known as self-training. The intuition behind this approach is to leverage the large amount of unlabelled data to improve the model's performance. Let’s break down whether the second model can add value and why this approach might be used, with references to relevant research to support the discussion. There are several reasons to use the second model:
Label Propagation and Decision Boundary Refinement:
When the first model predicts labels for the unlabelled data, it expands the labelled dataset. The second model can then use this expanded dataset to refine its decision boundaries. By training on more data points, the second model can learn more generalised patterns that the first model might not capture due to the limited labelled data. This concept is foundational and was demonstrated by Yarowsky (1995), who showed how leveraging unlabelled data could significantly enhance learning in natural language processing tasks.
Confidence in Predictions:
The first model might make correct predictions on the unlabelled data, adding useful information to the training process of the second model. Techniques like confidence thresholding can be used to only include predictions the first model is most confident in, thereby improving the quality of the pseudo-labels and reducing the risk of error propagation. Sohn et al (2020) with FixMatch (a semi-supervised learning method that simplifies and improves upon previous approaches by focusing on consistency regularisation and confidence thresholding) emphasises the importance of consistency and confidence in pseudo-labels, showing that models trained with high-confidence pseudo-labels on augmented data can achieve remarkable performance improvements.
Iterative Improvement:
This process can be iterative. The second model, after being trained on a mix of real and pseudo-labeled data, can generate better pseudo-labels for another iteration, and so on. Each iteration can potentially improve the quality of pseudo-labels, leading to better overall model performance. Berthelot et al. (2019) presented MixMatch (a holistic approach combining multiple techniques such as augmentation and mixing labelled and unlabelled data, to iteratively improve model accuracy in semi-supervised settings).
In theory, the second model can provide more accurate predictions if the process is managed correctly:
Noise Reduction:
The second model might reduce noise by training on a larger, more diverse dataset. Although some pseudo-labels might be incorrect, the larger dataset can help the model learn better overall patterns. This is aligned with the principles outlined by Xie et al. (2020) on Unsupervised Data Augmentation, which demonstrated the effectiveness of consistency training with augmented data to enhance model robustness and accuracy.
Regularisation Effect:
Training on a larger dataset, even with some noise, can have a regularisation effect, preventing the model from overfitting to the small labelled dataset and potentially improving its generalisation to new data. Chapelle, Scholkopf, and Zien (2006) provided a comprehensive overview of these semi-supervised learning techniques, illustrating how additional unlabeled data, when used appropriately, can regularise and improve model performance.
Of course, there are potential drawbacks:
Propagation of Errors:
Incorrect pseudo-labels can propagate errors, leading to a degradation in performance. To mitigate this, confidence thresholding can be employed to only use high-confidence pseudo-labels. Additionally, consistency regularisation, where models are trained to produce consistent predictions under small perturbations, can help improve robustness. The work by Sohn et al. (2020) with FixMatch underscores the efficacy of these methods in maintaining high-quality pseudo-labels.
Overconfidence:
The second model might become overconfident, especially if it heavily relies on incorrect pseudo-labels. Calibration techniques, such as Platt scaling or isotonic regression, can be used to address overconfidence and improve the reliability of the model's predictions.
Conclusion
In practice, self-training and other semi-supervised learning techniques have been shown to improve performance in various tasks, particularly when labeled data is scarce, and unlabeled data is abundant. The key is in the careful management and iterative refinement of pseudo-labels. Foundational work by Yarowsky (1995), along with more recent advancements by Berthelot et al. (2019) and Sohn et al. (2020), as well as comprehensive overviews like those by Chapelle, Scholkopf, and Zien (2006), and innovative approaches by Xie et al. (2020), provide strong evidence of the efficacy of these methods in improving model performance.
References
Berthelot, D., Carlini, N., Cubuk, E. D., Kurakin, A., Sohn, K., Zhang, H., & Raffel, C. (2019). MixMatch: A Holistic Approach to Semi-Supervised Learning. In Advances in Neural Information Processing Systems (pp. 5049-5059).
Chapelle, O., Scholkopf, B., & Zien, A. (2006). Semi-Supervised Learning. MIT Press.
Sohn, K., Berthelot, D., Li, C. L., Zhang, Z., Carlini, N., Cubuk, E. D., Kurakin, A., Zhang, H., & Raffel, C. (2020). FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence. In Advances in Neural Information Processing Systems (pp. 596-608).
Xie, Q., Dai, Z., Hovy, E., Luong, M.-T., & Le, Q. V. (2020). Unsupervised Data Augmentation for Consistency Training. In Advances in Neural Information Processing Systems (pp. 6256-6266).
Yarowsky, D. (1995). Unsupervised Word Sense Disambiguation Rivaling Supervised Methods. In Proceedings of the 33rd Annual Meeting on Association for Computational Linguistics (pp. 189-196).
