5
$\begingroup$

Suppose in semi-supervised learning, we have labeled set $X_L$ and unlabeled set $X_U$

  1. Is it ok to validate model performance on labeled data only?
  2. How to do cross-validation in transductive learning, in which the test set is $X_U$, which can also be considered as a part of the training data?
  3. While performing label propagation or graph neural networks on graph-based data, should we split all nodes randomly, or preprocess(like finding connected components) first?
$\endgroup$

1 Answer 1

2
$\begingroup$

This is a good question, and is in an area where we need more research. I believe that typically cross-validation or bootstrap resampling do not require inclusion of unsupervised learning steps in repeated loop. What is the result of unsupervised learning (principal components analysis being a prime example) can be unstable, overfitted, and overinterpreted. But this instability is in a random direction with regard to the response Y being predicted. In other words, you may benefit from undersupervised learning, but you can also be hurt by it. So it is not biased one way or another as it does not optimize prediction of Y. It tries to optimize how predictors relate to each other.

It is a good idea to bootstrap patterns learning in unsupervised learning (data reduction) to learn about the stability, doing this separately from the Y prediction validation. And there are some cases where to be confidence in the final predictions you want to include from-scratch data reduction inside the same loop as that used to validate predictive accuracy.

$\endgroup$
5
  • $\begingroup$ I would include the unsupervised steps in each fold of the cross-validation as the variance in the performance due to the unsupervised step should be a component of the uncertainty of the overall performance estimate. $\endgroup$ Commented Jun 16, 2022 at 14:27
  • 1
    $\begingroup$ A new paper was just published that is related to this: rss.onlinelibrary.wiley.com/doi/10.1111/rssb.12537?af=R $\endgroup$ Commented Jun 16, 2022 at 20:42
  • $\begingroup$ thanks, that looks like a very interesting/useful paper! $\endgroup$ Commented Jun 16, 2022 at 22:18
  • 1
    $\begingroup$ In terms of inclusion of the variance from the unsupervised step there are instabilities in unsupervised learning that don't bias model validation, and instabilities that might bias it. I'd like to better understand how to distinguish the two. $\endgroup$ Commented Jun 16, 2022 at 22:53
  • 1
    $\begingroup$ Yes, me too. Model selection is a particularly important topic for ML as the models often have a lot of hyper-parameters or structural choices (especially things like deep neural nets) and automated model search is popular. I prefer the safer approach and view CV as estimating the performance of a method of generating a model rather than of the final model itself, and that means repeating the whole procedure in each fold. Sometimes it is overkill though. $\endgroup$ Commented Jun 17, 2022 at 0:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.