1
$\begingroup$

I have an certain labeled data set suitable for a classification task. A classifier was optimized on this labeled data.

Time after this I received a very large set of the same type of data, albeit unlabeled.

I can classify all this data without issues using the previous classifier. But the data is too large to check if the classifier performs as well as estimated at the model building stage.

I would like to devise a sampling scheme to select a certain amount of observations to be labeled manually so as to estimate the classifiers accuracy per class on this new data set, also taking into account the classifiers "confidence". I failed to find examples of this on google so I would be looking for ideas and bibliography on the topic.

$\endgroup$
  • 1
    $\begingroup$ Many methods assume relatively balanced number of instances per class. Sampling like that is often called stratified. Apart from that, I don't see why not random sample? As many as you can afford... $\endgroup$ – jonnor May 26 '18 at 23:24
  • $\begingroup$ It makes sense to me to stratify based on each class, would you use the size of each class on the new set with imputed labels to choose the sample size within strata? do you by any chance know of some work were this has been performed? Based on this how would you study the confusion between classes? $\endgroup$ – JEquihua May 26 '18 at 23:30
  • 1
    $\begingroup$ If the class distribution is not too bad, just random sample without worrying about stratification. If it does need to be accounted for, classify new data with existing model, then random sample for each predicted class and send for labelling to establish actual ground truth labels? Make sure labeling process is blind. I suggest same size per class, unless labeling is too expensive and it is horribly imbalanced. Then adjust based on class probability from original training set (hopefully a decent estimate for new set, assuming no distribution shifts) $\endgroup$ – jonnor May 26 '18 at 23:55
  • $\begingroup$ Somehow there seems to be something missing. How could you check the classification on new data? If you have all the information inside the data itself, it seems a bit odd to use something else; especially something that you had to build before. Maybe there are parts missing in the description -- why do you need a classifier at all? What are the constraints of validation, what do you mean by 'too large'? $\endgroup$ – cherub May 29 '18 at 9:33
2
+100
$\begingroup$

First of all: to derive a strategy that is better than pure random sampling, a lot of further input/knowledge about the application is needed, and you need to decide and specify what aspects of performance you want to measure.

Some considerations:

  • E.g. the positive/negative predictive values are important figures of merit, and you can access them comparatively easily in your case: select the decided number of cases for reference labeling from the postively and negatively predicted cases.

  • Likewise, you may want to establish performance for borderline cases by examining not-so-clear-cut predictions.

  • If you do any systematic selection among your predictions, you need to take these sampling schemes into account when calculating general figures of merit (i.e. weight appropriately).
    Keep in mind any stratified sampling you can do at the moment will stratify predicted class while the "usual" stratification when planning data acquisition is stratified reference class membership.

  • You'd want to know and take into account the process that generated your large data set. E.g. consider whether there are possible confounders that lead to clusters in the large data? => in that case you may want to systematically probe that confounder.

  • Don't forget that test sample size determines (un)certainty of your figures of merit. E.g. for a proportion like PPV or NPV, as a rule of thumb you'll need 100 cases in the denominator (i.e. positively predicted cases) to get a 95% confidence interval that is about 10%points wide.
    Beleites, C. and Neugebauer, U. and Bocklitz, T. and Krafft, C. and Popp, J.: Sample size planning for classification models. Anal Chim Acta, 2013, 760, 25-33.
    DOI: 10.1016/j.aca.2012.11.007

    accepted manuscript on arXiv: 1211.1323

  • taking into account the classifiers "confidence"

    Have a look at literature about scoring rules. Scoring rules measure whether predicted probabilities (scores) for class membership are correct.
    E.g., if you take all cases where your model predicts 90 % probability for class A then 90 % of them should really belong to class A.

    Compared to "hard" proportions like PPV, NPV, accuracy etc. the advantage of scoring rules is that they react "soft" and continuously. I.e. a slightly lower predicted probability will lead to a small loss whereas e.g. accuracy would not detect such a small deterioration unless the cutoff threshold is crossed: then a slight change would be counted as full error (and would not be distinguishable from a misclassification where the classifier was "certain" of the wrong class).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.