Model selection and error estimation with active learning

Question

I'm working on a classification problem where we have plenty of unlabelled data but it is costly and time-consuming to label them. So, I'm looking into active learning to economize on how much data we need to label.

What I have not been able to figure out yet, is how model selection and error estimation works with active learning. Given that a large set of labelled examples is not available, there are no held-out validation or test sets. So, it appears that one cannot compare different models (model selection) or estimate the generalization error of a given model.

Is this so? How do people go about model selection and error estimation when using active learning?

I'd appreciate any comments, explanations or relevant references.

Answer 1

The principals of model selection are the same when using active learning as with standard classification. You still want to make model selection decisions using separate data (e.g. validation set or cross-validation) from the training data to avoid overfitting of the hyperparameters. In the case where labeled data is very rare, leave-one-out cross-validation may be a good option because you only leave out one example with each CV fold and the computational costs are manageable for small training sets. Use simple models and don't expect to do much hyperparameter optimization or model selection until you can gather more labeled examples.

There is some discussion in this article. Also see "Example 1" which discusses failures of leave-one-out cross validation for the small Iris dataset. http://web.cs.iastate.edu/~jtian/cs573/Papers/Kohavi-IJCAI-95.pdf

You may also consider other techniques that help when working with small training sets: semi-supervised methods, bootstrap sampling, look for data augmentation opportunities, or model-based machine learning.

https://medium.com/rants-on-machine-learning/what-to-do-with-small-data-d253254d1a89 https://blog.dominodatalab.com/an-introduction-to-model-based-machine-learning/