Semi-supervised metric learning?

Question

I was reading an article about metric learning lately. http://arxiv.org/pdf/1306.6709.pdf. In the paper, the author indicates that there are three types of metric learning paradigms, i.e., the full supervised learning, the weak supervised learning and the semi-supervised learning.

What I was confused a bit is that, the author mentioned that for the semi-supervised learning, "besides the full or weak supervision, the algorithm has access to a typically large sample of unlabeled instances for which no side information is available. This is typically useful to avoid overfitting when the labeled data or side information is scarce". So, can anyone explain to me why the unlabeled instances would help to avoid overfitting?

Answer 1

If there is lots of parameters to learn, it would be hard to learn them with small size of data. Instead, unsupervised learning is employed on big unlabeled data to reveal common features that can be useful to represent the data with small size. For example, features for natural images can be learned on unlabeled natural scenes and this features can be used to reveal structures on another data set.

There is two way to implement this as far as I know.

1. Use features trained on unlabeled dataset to reveal abstract features on your small labeled data. Then use this abstract feature to learn a classifier like soft-max or SVM. In that case, small dataset is only used to optimize parameters of the classifier (not sure if SVM has a parameter at all but that is the case for soft-max or other logistic type of classifiers). As a result of reduced parameters, over-fitting is reduced.

2. The first methods is a good approach. However, it is not fine-tuned which means the features learnt on another dataset is good, to reveal structures in the input data, but it can be better if they are well adapted to the current set (small set). In this type, pre-training (training on another dataset) is used for initialization and it introduce good regularization for the training algorithm (check the paper below). Even though number of parameters is not reduced with respect to the first model, good initialization make it more likely to converge good local minima since it is already close good it. Without pre-training and good initialization, there is too much degree of freedom to converge a good minima.

Here is paper that explains this concept in detail on Neural Networks. Shorter version, longer version