1
$\begingroup$

With semi-supervised learning a labeled set $X_L$ and unlabeled set $X_U$ are given. If the learning algorithm has several free-parameters we are forced to perform cross-validation to try to guess them. Cross-validation can only be applied to the labeled set. So:

  1. If we have very few labeled examples, say about 1%-10%, is it better to apply LOO-CV?

  2. If the ratio of labeled samples is bigger (e.g. 50-60%), k-fold CV might be better (as LOO-CV can introduce a huge bias), but then, can we assume that the sets will be $i.i.d.$? Here is it better to pick a low k?

What is the best way validate a model in semi-supervised learning?

Improve this question
$\endgroup$
0
0
$\begingroup$

Cross-validation procedures are not invalidated in this setting, provided you use a suitable score function.

A suitable score function could be recall^2 divided by the probability for a classifier to predict one. This is a surrogate for the F1-score in a PU learning setting. You can find an example of its use here.

Improve this answer
$\endgroup$
6
  • $\begingroup$ I understand that recall is calculated using the labeled validation set and probability to predict class using labeled and unlabeled examples. How can I estimate the probability for a classifier to predict a class? Using simple probabilities such as $p_i = class_i/total$ would be okey? $\endgroup$ – user45299 Dec 5 '14 at 23:02
  • $\begingroup$ @KoTy yes, that is the only way to do it. It is a very crude approximation, but it's the best you can do with what little information you have. $\endgroup$ – Marc Claesen Dec 5 '14 at 23:24
  • $\begingroup$ If I use ove-vs-all trainig how can I estimate the probability to predict one? It is possible to calculate recall and precision as they need TP, FN and FP, but how do I use the probability? $\endgroup$ – user45299 Dec 5 '14 at 23:53
  • $\begingroup$ Besides, when doing cross-validation the probability of predicting one may change in semi-supervised. How to choose a measure then? $\endgroup$ – user45299 Dec 6 '14 at 0:15
  • $\begingroup$ @KoTy when calculating the given score (per fold), you basically assume the entire unlabeled set is negative (which is a conservative approximation) and compute the probability of positive prediction as the number of positive predictions over the fold size. $\endgroup$ – Marc Claesen Dec 6 '14 at 22:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy