1
$\begingroup$

With semi-supervised learning a labeled set $X_L$ and unlabeled set $X_U$ are given. If the learning algorithm has several free-parameters we are forced to perform cross-validation to try to guess them. Cross-validation can only be applied to the labeled set. So:

  1. If we have very few labeled examples, say about 1%-10%, is it better to apply LOO-CV?

  2. If the ratio of labeled samples is bigger (e.g. 50-60%), k-fold CV might be better (as LOO-CV can introduce a huge bias), but then, can we assume that the sets will be $i.i.d.$? Here is it better to pick a low k?

What is the best way validate a model in semi-supervised learning?

|cite|improve this question
$\endgroup$
1
$\begingroup$

Cross-validation procedures are not invalidated in this setting, provided you use a suitable score function.

A suitable score function could be recall^2 divided by the probability for a classifier to predict one. This is a surrogate for the F1-score in a PU learning setting. You can find an example of its use here.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ I understand that recall is calculated using the labeled validation set and probability to predict class using labeled and unlabeled examples. How can I estimate the probability for a classifier to predict a class? Using simple probabilities such as $p_i = class_i/total$ would be okey? $\endgroup$ – KoTy Dec 5 '14 at 23:02
  • $\begingroup$ @KoTy yes, that is the only way to do it. It is a very crude approximation, but it's the best you can do with what little information you have. $\endgroup$ – Marc Claesen Dec 5 '14 at 23:24
  • $\begingroup$ If I use ove-vs-all trainig how can I estimate the probability to predict one? It is possible to calculate recall and precision as they need TP, FN and FP, but how do I use the probability? $\endgroup$ – KoTy Dec 5 '14 at 23:53
  • $\begingroup$ Besides, when doing cross-validation the probability of predicting one may change in semi-supervised. How to choose a measure then? $\endgroup$ – KoTy Dec 6 '14 at 0:15
  • $\begingroup$ @KoTy when calculating the given score (per fold), you basically assume the entire unlabeled set is negative (which is a conservative approximation) and compute the probability of positive prediction as the number of positive predictions over the fold size. $\endgroup$ – Marc Claesen Dec 6 '14 at 22:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.