Performance evaluation with non-representative data I am currently trying to apply some models for text classification on a binary task. The core two approaches that I follow are, on the one hand, using word2vec vector representations on a Random Forest and, on the other hand, using a fined tuned BERT.
My issue is that my training data has been labelled by me in some sort of active learning fashion: I started labelling some examples, then checked the performance on the top classification, relabel and feed back into the training data. After several rounds, this process led me with a dataset in which the class unbalancedness is definitely not even close to the one that the algorithms actually see when I feed in new data. To give you some quick numbers, the proportion of 1s on my training data is around 40% whereas the one resulting from a normal classification can be around 10% or lower.
This interferes with my performance testing. Since I test different metrics on a test set which is way less imbalanced than the "real" distribution, I can't be sure that this is indeed a valid procedure to do comparisons across models. Augmenting the training data to make it more realistic is technically infeasible ( I would need to classify way too many examples atm).
Is there any method to produce reliable performance metrics when this happens? I thought about synthetic sampling to augment the volume of 0s but I fear this might just be producing similar results...
EDIT: Scheme of the process:

Train and test set are taken from the labelled data with class distribution 40-60%. This does not reflect the distribution of the unlabelled data (much more imbalanced), and hence evaluation metrics can't be trusted since this is not what the algorithm faces when it has to do predictions.
 A: If you had oversampled one class randomly (e.g. if instead of 10% of examples in some class you had picked 50% from that class), this would be easy to deal with. You'd simply have to adjust metrics accordingly. E.g. let's say you oversampled class 1, you want some confusion matrix based metrics, and the numbers in a confusion matrix are like this:




True class
Predicted class 1
Predicted class 2
Totals




1
90
10
100 (50%)


2
10
90
100 (50%)




Then, you'd create a modified confusion matrix like this:




True class
Predicted class 1
Predicted class 2
Totals




1
9.99...
1.11...
11.11... (10%)


2
10
90
100 (90%)




And then you calculate metrics based on this. Of course, you could also do this through some appropriate models (e.g. for accuracy a logistic regression with an offset). Obviously, this would mess with any confidence intervals you'd want to create so that would be harder to get.
However, you did not sample randomly and it's hard to see any realistic way you could adjust for your sampling method due to its rather ad-hoc nature. Sampling methods that can be described in a model (e.g. random sampling), you can deal with through some modeling to get metrics. Active learning may or may not be perfectly fine to do, but you should only do it on your training data, not on your evaluation data. Ideally, you'd not even want to oversample on your evaluation data, but at least if you do, you could correct for it (some metrics, as you say, will be heavily skewed by oversampling some class).
A: As @Björn explains, your test set is biased because you created it by deterministically choosing the "top examples" to label. This introduces bias in the evaluation that you cannot correct for.
However, you don't necessarily need to sample examples randomly to evaluate machine learning models correctly if you are more strategic in how you select examples to label.
This is called active testing; you can learn more about it in [1].
The gist: It's okay to introduce bias by actively selecting test examples to label as long as it is possible to remove the bias afterwards. Obviously this makes for more efficient evaluation; it can also reduce the variance of the performance estimator.
The procedure outlined in [1] uses a surrogate model to estimate acquisition probabilities, which indicate how valuable an example is to label next. So instead of choosing an example deterministically, you sample it according to its acquisition probability. The idea behind using the surrogate is to (a) account for uncertainty over the outcomes; (b) make predictions that are diverse to the model f(x) under evaluation, and (c) incorporate information from all available data.
[1] J. Kossen, S. Farquhar, Y. Gal, and T. Rainforth. Active testing: Sample-efficient model evaluation. In Proceedings of the 38th International Conference on Machine Learning, 2021. https://doi.org/10.48550/arXiv.2103.05331
