7
$\begingroup$

I am confused with the concepts of stratified sampling and oversampling for imbalanced datasets. From what I read in this question here: why-use-stratified-cross-validation-why-does-this-not-damage-variance-related , stratified sampling helps to make sure the sample contains the sample proportion of each class, so that the sample is representative.

But in many other posts I have read that oversampling rare events for imbalanced datasets is some helpful technique to handle high false positive or false negative problems.

Aren't the 2 sampling ideas contrary with each other? As in one is to make sure the sample is representative of the population, while the other is balancing the sample classes manually? I know I might confuse some fundamental concepts here. Can anyone help?

$\endgroup$
1

1 Answer 1

2
$\begingroup$

To a large extent, the desire to apply artificial balancing comes from using improper scoring rules, chiefly accuracy.

In particular, it seems that people realized that a model with strong imbalance could achieve an impressive-looking $98\%$ accuracy by predicting yet underperform a model that always predicts the majority category, such as if the majority category represents $99\%$ of all observations. Consequently, people seem to have changed the class ratio to allow for such a high accuracy to be more reflective of strong performance: if you balance the classes, then predicting one class every time results in $50\%$ accuracy (or worse, if there or three or more classes), so scoring $98\%$ would be quite an improvement.

I see this as a major drawback of accuracy, and a simple remedy, comparison of error rates, makes it more comparable to $R^2$ in regression and might be a more useful measure of performance. I show in the link what happens when you have an accuracy score than looks high but underperforms predicting the majority class every time, and this statistic indeed flags that as poor performance.

However, accuracy, comparison of error rates, and some classics like sensitivity, specificity, and $F_1$ score, all have the downside of being based on hard classifications. Most "classification" models, such a logistic regressions and neural networks, do not output predicted class labels. Instead, they output values on a continuum that often can be interpreted as a probability, and then you can make decisions based on those probabilities. Importantly, those decisions can depend on factors other than the probabilities (such a features in the model...might be more willing to make a certain decision for the usual people than with a high-roller) and can be more numerous than the categories.

There are exceptions to the upcoming statement, such as in data collection or perhaps for computational reasons when it comes to numerical optimization of neural networks, but the apparent problems when it comes to class imbalance largely do not manifest when models are evaluated on the continuous predictions.

You are correct to point out that representative samples and oversampling contradict each other, but oversampling is largely a solution to a non-problem. For the most part, I am with you that it makes sense to develop models on representative data. If a category is rare, then we should be skeptical that an observation belongs to it by assigning a low prior probability and making the features have to shine through to prove that there is a strong chance that the observation indeed belongs to that category.

This link is already elsewhere in this answer, but many of the claimed issues with class imbalance are debunked here by our Stephan Kolassa.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.