I was reading an article about metric learning lately. http://arxiv.org/pdf/1306.6709.pdf. In the paper, the author indicates that there are three types of metric learning paradigms, i.e., the full supervised learning, the weak supervised learning and the semi-supervised learning.

What I was confused a bit is that, the author mentioned that for the semi-supervised learning, "besides the full or weak supervision, the algorithm has access to a typically large sample of unlabeled instances for which no side information is available. This is typically useful to avoid overfitting when the labeled data or side information is scarce". So, can anyone explain to me why the unlabeled instances would help to avoid overfitting?


If there is lots of parameters to learn, it would be hard to learn them with small size of data. Instead, unsupervised learning is employed on big unlabeled data to reveal common features that can be useful to represent the data with small size. For example, features for natural images can be learned on unlabeled natural scenes and this features can be used to reveal structures on another data set.

There is two way to implement this as far as I know.

  1. Use features trained on unlabeled dataset to reveal abstract features on your small labeled data. Then use this abstract feature to learn a classifier like soft-max or SVM. In that case, small dataset is only used to optimize parameters of the classifier (not sure if SVM has a parameter at all but that is the case for soft-max or other logistic type of classifiers). As a result of reduced parameters, over-fitting is reduced.

  2. The first methods is a good approach. However, it is not fine-tuned which means the features learnt on another dataset is good, to reveal structures in the input data, but it can be better if they are well adapted to the current set (small set). In this type, pre-training (training on another dataset) is used for initialization and it introduce good regularization for the training algorithm (check the paper below). Even though number of parameters is not reduced with respect to the first model, good initialization make it more likely to converge good local minima since it is already close good it. Without pre-training and good initialization, there is too much degree of freedom to converge a good minima.

Here is paper that explains this concept in detail on Neural Networks. Shorter version, longer version

  • 1
    $\begingroup$ It's also common to use unsupervised data for a regularization term: you search for a model that simultaneously explains the labeled data well, and is sufficiently smooth (or whatever) on unlabeled data. $\endgroup$ – Dougal Jun 11 '15 at 21:45
  • 1
    $\begingroup$ @Dougal Could you elaborate more onto that or provide a paper? I would like to learn it further. $\endgroup$ – yasin.yazici Jun 11 '15 at 21:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.