Suppose in semi-supervised learning, we have labeled set $X_L$ and unlabeled set $X_U$

  1. Is it ok to validate model performance on labeled data only?
  2. How to do cross-validation in transductive learning, in which the test set is $X_U$, which can also be considered as a part of the training data?
  3. While performing label propagation or graph neural networks on graph-based data, should we split all nodes randomly, or preprocess(like finding connected components) first?

This is a good question, and is in an area where we need more research. I believe that typically cross-validation or bootstrap resampling do not require inclusion of unsupervised learning steps in repeated loop. What is the result of unsupervised learning (principal components analysis being a prime example) can be unstable, overfitted, and overinterpreted. But this instability is in a random direction with regard to the response Y being predicted. In other words, you may benefit from undersupervised learning, but you can also be hurt by it. So it is not biased one way or another as it does not optimize prediction of Y. It tries to optimize how predictors relate to each other.

It is a good idea to bootstrap patterns learning in unsupervised learning (data reduction) to learn about the stability, doing this separately from the Y prediction validation. And there are some cases where to be confidence in the final predictions you want to include from-scratch data reduction inside the same loop as that used to validate predictive accuracy.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.