I'm working on a classification problem where we have plenty of unlabelled data but it is costly and time-consuming to label them. So, I'm looking into active learning to economize on how much data we need to label.
What I have not been able to figure out yet, is how model selection and error estimation works with active learning. Given that a large set of labelled examples is not available, there are no held-out validation or test sets. So, it appears that one cannot compare different models (model selection) or estimate the generalization error of a given model.
Is this so? How do people go about model selection and error estimation when using active learning?
I'd appreciate any comments, explanations or relevant references.