Cross validation in semi-supervised learning

With semi-supervised learning a labeled set $X_L$ and unlabeled set $X_U$ are given. If the learning algorithm has several free-parameters we are forced to perform cross-validation to try to guess them. Cross-validation can only be applied to the labeled set. So:

1. If we have very few labeled examples, say about 1%-10%, is it better to apply LOO-CV?

2. If the ratio of labeled samples is bigger (e.g. 50-60%), k-fold CV might be better (as LOO-CV can introduce a huge bias), but then, can we assume that the sets will be $i.i.d.$? Here is it better to pick a low k?

What is the best way validate a model in semi-supervised learning?

Cross-validation procedures are not invalidated in this setting, provided you use a suitable score function.

A suitable score function could be recall^2 divided by the probability for a classifier to predict one. This is a surrogate for the F1-score in a PU learning setting. You can find an example of its use here.

• I understand that recall is calculated using the labeled validation set and probability to predict class using labeled and unlabeled examples. How can I estimate the probability for a classifier to predict a class? Using simple probabilities such as $p_i = class_i/total$ would be okey?
– user45299
Dec 5, 2014 at 23:02
• @KoTy yes, that is the only way to do it. It is a very crude approximation, but it's the best you can do with what little information you have. Dec 5, 2014 at 23:24
• If I use ove-vs-all trainig how can I estimate the probability to predict one? It is possible to calculate recall and precision as they need TP, FN and FP, but how do I use the probability?
– user45299
Dec 5, 2014 at 23:53
• Besides, when doing cross-validation the probability of predicting one may change in semi-supervised. How to choose a measure then?
– user45299
Dec 6, 2014 at 0:15
• @KoTy when calculating the given score (per fold), you basically assume the entire unlabeled set is negative (which is a conservative approximation) and compute the probability of positive prediction as the number of positive predictions over the fold size. Dec 6, 2014 at 22:51