I am currently trying to apply some models for text classification on a binary task. The core two approaches that I follow are, on the one hand, using word2vec vector representations on a Random Forest and, on the other hand, using a fined tuned BERT.

My issue is that my training data has been labelled by me in some sort of active learning fashion: I started labelling some examples, then checked the performance on the top classification, relabel and feed back into the training data. After several rounds, this process led me with a dataset in which the class unbalancedness is definitely not even close to the one that the algorithms actually see when I feed in new data. To give you some quick numbers, the proportion of 1s on my training data is around 40% whereas the one resulting from a normal classification can be around 10% or lower.

This interferes with my performance testing. Since I test different metrics on a test set which is way less imbalanced than the "real" distribution, I can't be sure that this is indeed a valid procedure to do comparisons across models. Augmenting the training data to make it more realistic is technically infeasible ( I would need to classify way too many examples atm).

Is there any method to produce reliable performance metrics when this happens? I thought about synthetic sampling to augment the volume of 0s but I fear this might just be producing similar results...

EDIT: Scheme of the process:

Train and test set are taken from the labelled data with class distribution 40-60%. This does not reflect the distribution of the unlabelled data (much more imbalanced), and hence evaluation metrics can't be trusted since this is not what the algorithm faces when it has to do predictions.


2 Answers 2


If you had oversampled one class randomly (e.g. if instead of 10% of examples in some class you had picked 50% from that class), this would be easy to deal with. You'd simply have to adjust metrics accordingly. E.g. let's say you oversampled class 1, you want some confusion matrix based metrics, and the numbers in a confusion matrix are like this:

True class Predicted class 1 Predicted class 2 Totals
1 90 10 100 (50%)
2 10 90 100 (50%)

Then, you'd create a modified confusion matrix like this:

True class Predicted class 1 Predicted class 2 Totals
1 9.99... 1.11... 11.11... (10%)
2 10 90 100 (90%)

And then you calculate metrics based on this. Of course, you could also do this through some appropriate models (e.g. for accuracy a logistic regression with an offset). Obviously, this would mess with any confidence intervals you'd want to create so that would be harder to get.

However, you did not sample randomly and it's hard to see any realistic way you could adjust for your sampling method due to its rather ad-hoc nature. Sampling methods that can be described in a model (e.g. random sampling), you can deal with through some modeling to get metrics. Active learning may or may not be perfectly fine to do, but you should only do it on your training data, not on your evaluation data. Ideally, you'd not even want to oversample on your evaluation data, but at least if you do, you could correct for it (some metrics, as you say, will be heavily skewed by oversampling some class).

  • $\begingroup$ Thanks a lot for the detailed comment! My sampling/labeling method is not random at all. Perhaps I explained myself wrongly on my first post: All my labeled data comes from an active training procedure (I started from scratch) where I classify mostly the 1st class predictions. From this labelled data I split into training-test to do performance evaluation but the class imbalancednes is way different from the one in the non-labelled data. From your last comment I get that the solution might be to oversample on the test data to make it more realistic? $\endgroup$
    – Luis
    Jun 15, 2022 at 14:41
  • $\begingroup$ I edited the original post with a quick scheme. I hope it illustrates the explanation. $\endgroup$
    – Luis
    Jun 15, 2022 at 15:03
  • $\begingroup$ My point is that there are solutions, if you had (which you haven't) sampled randomly in some form. Given what you did, I don't think there would be any realistic way to use data generated in this manner for evaluation (even if we assume that the original set of data perfectly represents your true target distribution) so that this comes close to how the real data behave. To get a decent evaluation of that, you'd want a new set of data without such labelling. Even taking some of the not yet labelled data from the current set as the starting point may be problematic. $\endgroup$
    – Björn
    Jun 15, 2022 at 15:20
  • $\begingroup$ Okay I get it know. I might ten consider classifying some new data preserving the class proportions using random sampling. I guess that might be the easiest way around. $\endgroup$
    – Luis
    Jun 16, 2022 at 8:25

As @Björn explains, your test set is biased because you created it by deterministically choosing the "top examples" to label. This introduces bias in the evaluation that you cannot correct for.

However, you don't necessarily need to sample examples randomly to evaluate machine learning models correctly if you are more strategic in how you select examples to label.

This is called active testing; you can learn more about it in [1].

The gist: It's okay to introduce bias by actively selecting test examples to label as long as it is possible to remove the bias afterwards. Obviously this makes for more efficient evaluation; it can also reduce the variance of the performance estimator.

The procedure outlined in [1] uses a surrogate model to estimate acquisition probabilities, which indicate how valuable an example is to label next. So instead of choosing an example deterministically, you sample it according to its acquisition probability. The idea behind using the surrogate is to (a) account for uncertainty over the outcomes; (b) make predictions that are diverse to the model f(x) under evaluation, and (c) incorporate information from all available data.

[1] J. Kossen, S. Farquhar, Y. Gal, and T. Rainforth. Active testing: Sample-efficient model evaluation. In Proceedings of the 38th International Conference on Machine Learning, 2021. https://doi.org/10.48550/arXiv.2103.05331


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.