Sampling design to measure the performance of a classifier on large set of new data I have an certain labeled data set suitable for a classification task. A classifier was optimized on this labeled data. 
Time after this I received a very large set of the same type of data, albeit unlabeled. 
I can classify all this data without issues using the previous classifier. But the data is too large to check if the classifier performs as well as estimated at the model building stage.
I would like to devise a sampling scheme to select a certain amount of observations to be labeled manually so as to estimate the classifiers accuracy per class on this new data set, also taking into account the classifiers "confidence". I failed to find examples of this on google so I would be looking for ideas and bibliography on the topic.
 A: First of all: to derive a strategy that is better than pure random sampling, a lot of further input/knowledge about the application is needed, and you need to decide and specify what aspects of performance you want to measure.
Some considerations:


*

*E.g. the positive/negative predictive values are important figures of merit, and you can access them comparatively easily in your case: select the decided number of cases for reference labeling from the postively and negatively predicted cases. 

*Likewise, you may want to establish performance for borderline cases by examining not-so-clear-cut predictions. 

*If you do any systematic selection among your predictions, you need to take these sampling schemes into account when calculating general figures of merit (i.e. weight appropriately).
Keep in mind any stratified sampling you can do at the moment will stratify predicted class while the "usual" stratification when planning data acquisition is stratified reference class membership. 

*You'd want to know and take into account the process that generated your large data set. E.g. consider whether there are possible confounders that lead to clusters in the large data? => in that case you may want to systematically probe that confounder.

*Don't forget that test sample size determines (un)certainty of your figures of merit. E.g. for a proportion like PPV or NPV, as a rule of thumb you'll need 100 cases in the denominator (i.e. positively predicted cases) to get a 95% confidence interval that is about 10%points wide.
Beleites, C. and Neugebauer, U. and Bocklitz, T. and Krafft, C. and Popp, J.: Sample size planning for classification models. Anal Chim Acta, 2013, 760, 25-33.
DOI: 10.1016/j.aca.2012.11.007
accepted manuscript on arXiv: 1211.1323

*
taking into account the classifiers "confidence"  

Have a look at literature about scoring rules. Scoring rules measure whether predicted probabilities (scores) for class membership are correct.
E.g., if you take all cases where your model predicts 90 % probability for class A then 90 % of them should really belong to class A. 
Compared to "hard" proportions like PPV, NPV, accuracy etc. the advantage of scoring rules is that they react "soft" and continuously. I.e. a slightly lower predicted probability will lead to a small loss whereas e.g. accuracy would not detect such a small deterioration unless the cutoff threshold is crossed: then a slight change would be counted as full error (and would not be distinguishable from a misclassification where the classifier was "certain" of the wrong class).
