Performing Data Analysis on Results of ML classifier applied to Unlabelled Data

Question

Suppose I have a large set of manually labelled data (e.g. 5000+ instances) with one of two lables, A or B, and I intend to build a ML classifier from this data. Using a proper methodology (e.g. cross validation, dividing data into training, testing, and validation sets etc.), suppose I obtain a classification accuracy (or ROC score) of ~85% on the testing set.

If I were to now take this classifier and let it loose on more instances (10,000+) of the exact same type of data, but this time the data has no labels (ie. ground truth), could I safely assume that 85% of these new instances are correctly labelled with respect to our ground truth training set (the original 5000+ instances)?

In addition, if I were to apply simple data analyses (e.g. t-tests) to these 10,000+ newly labelled instances, do there exist any methods to incorporate the fact that these instances may be only 85% correct? In other words is there a way to include in the error of the actual labelling of the data into the significance calculations for the distribution, counts, patterns etc. of the labels?

And last, does the scenario I'm describing sound like a situation that is better suited for semi-supervised learning methods?

Answer 1

If you have tested your model on an unseen test set and then did not tune again to improve results on the said test set, assuming that the test set is large enough then you will have a bias-free estimation of the error. When you apply your model to other unseen test sets, that come from the same data distribution, you can expect to have a similar performance.

About the confidence in your prediction, the output a classifier is usually a score, not a label. For example, in a neural network, the output is the probability of an instance belonging to a class. You can work with this information instead of the predicted labels.

Regarding semi-supervised learning, you could use the Expectation-Maximization algorithm to take advantage of the unlabeled data. In this algorithm, you repeatedly predict labels for the unlabeled data and then use the predicted labels to train a new model. Under some particular conditions, this actually improves performance on unseen data.