I have an certain labeled data set suitable for a classification task. A classifier was optimized on this labeled data.
Time after this I received a very large set of the same type of data, albeit unlabeled.
I can classify all this data without issues using the previous classifier. But the data is too large to check if the classifier performs as well as estimated at the model building stage.
I would like to devise a sampling scheme to select a certain amount of observations to be labeled manually so as to estimate the classifiers accuracy per class on this new data set, also taking into account the classifiers "confidence". I failed to find examples of this on google so I would be looking for ideas and bibliography on the topic.