2
$\begingroup$

I've been getting some conflicting advice from various ML podcasts/videos/articles lately for how to deal with imbalanced datasets. Let's say my independent variable for a classification problem has a 1000:1 ratio of negative:positive values.

Some people say that I should upweight the minority class here so that the model is penalized more for getting those predictions wrong. So maybe my weights could be {0: 1, 1: 100} or something like that

However, some other sources say that I should downsample the majority class and then upweight that same class - so in this case let's just say I keep 20% of the majority class, I would then train the model but with weights of {0: 20, 1: 1}

Is one of these "more right" than the other? Is the answer just to try both and see which gives better results? They just seem like such different approaches, I'm trying to figure out if I'm getting bad information somewhere.

$\endgroup$
5
  • 4
    $\begingroup$ Why would you throw away data? And why do you believe you need to do anything at all with it? $\endgroup$
    – Tim
    Commented Mar 31, 2022 at 12:49
  • 3
    $\begingroup$ fharrell.com/post/classification and fharrell.com/post/class-damage will be of interest. $\endgroup$
    – Dave
    Commented Mar 31, 2022 at 13:55
  • $\begingroup$ RE Tim: "why do you believe you need to do anything at all with it?" - I definitely think it makes sense to manipulate the data in some way to help a model make better predictions, otherwise the model will be biased to predict the negative flags "why would you throw away data?" - yeah I definitely would rather not throw away data. In the past, I've upsampled the minority class or used class weighting, this downsample + upweight majority class is a new concept to me. $\endgroup$
    – Peter M
    Commented Apr 1, 2022 at 13:03
  • $\begingroup$ RE Dave: thanks! $\endgroup$
    – Peter M
    Commented Apr 1, 2022 at 13:17
  • 2
    $\begingroup$ @PeterM The issues with class imbalance are (largely) due to using improper measures of performance. When you use the right statistical methods, which are discussed in the Frank Harrell blog I linked, the issues pretty much go away. $\endgroup$
    – Dave
    Commented Nov 13, 2022 at 18:29

3 Answers 3

1
$\begingroup$

Both approaches have minimal statistical motivation and seem to address a non-problem. (There is a very interesting case described in the answer to the linked question that relates to King & Zeng (2001), but I would argue that to be an issue of experimental design, rather than of model evaluation.)

Yes, class imbalance poses problems to the classification accuracy metric in that a score of $97\%$ might sound great but actually be quite pitiful if you would get $99.9\%$ of the cases correct by classifying as the majority class every time. However, this strikes me as a drawback of classification accuracy as a performance metric, rather than of the reality of your problem having imbalanced classes. (I discuss here and here how to remedy accuracy scores to deal with being high yet pitiful.)

Most machine learning methods "classifiers" give outputs on a continuum, and every method I know that does not can be wrestled with to give such a output on a continuum (e.g., Platt scaling for SVMs). Consequently, when you refer to the classification accuracy of a machine learning model, you either mean one of the following:

  1. Your model has (close to) $0\%$ accuracy, since dead-on predictions on the continuum are so unlikely.

  2. You are referring to the continuous predictions made by your model along with a decision rule that partitions that continuum into discrete buckets that make your classifications. The common decision rule is to classify predictions above $0.5$ as category $1$ and predictions below $0.5$ as category $0$.

When you do the second of the two, as many machine learning practitioners do and what happens when you call something like the predict method in software like sklearn, you are not actually evaluating the model. You are evaluating the model along with the decisions made using model outputs. However, especially if you give no thought to the decision rule, that decision rule might be terrible for your problem. Instead of fiddling with the data to remedy class imbalance issues, the first thought should be to change the decision rule. If you have $1000$:$1$ imbalance, maybe you want to predict as the minority class whenever the output (which often has an interpretation as a probability) is above $0.001$ instead of $0.5$. My logic for this is that, the baseline rate of minority-class occurrence is one-in-a-thousand, so if you even have a one-in-two-hundred chance of being in the minority class, that is a sizeable deviation from the norm and might warrant consideration. I discuss this idea in my question here.

The more sophisticated approach would be to consider the continuous outputs. In fact, doing so allows you to have more decisions that categories. It might be that your decision rule assigns predictions to category $0$ if the prediction is below $0.2$, assigns predictions to category $1$ if the prediction is above $0.9$, and gives an "I don't know" classification for predictions between $0.2$ and $0.9$, related to the idea presented here about putting "suspected spam" in an email subject line, rather than sending the message to the spam folder or letting it through without such a tag. We want confident and accurate predictions, sure, but those need not be realistic, and part of your job (or someone's job) is to handle that ambiguity.

All of this is to say that class imbalance is not inherently a problem. Good statistical methods like evaluating proper scoring rules, assessing the continuous outputs of models, and thinking in terms of event probability vs error cost handle class imbalance fine.

Consequently, it does not make much sense to fiddle with your data (downsampling) or the model-fitting process (weighting) to skew the outputs a certain way. Low predicted probabilities of unlikely outcomes seems like a feature, not a bug, of machine learning outputs. Fiddling with the statistics in order to get on the correct size of a software-default cutoff of $0.5$ seems like a poor approach to modeling when you consider what you lose by doing so.

Plenty of blogs and even sources that might seem more credible will advocate for these poor statistical methods because the authors are unaware of the statistical subtleties. It's a shame that our field has to fight this noise.

(Finally, downsampling strikes me as the worst of all approaches. Not only are you trying to solve something that is not a problem, but you are sacrificing precious data in order to do so. While upsampling, synthesizing points (e.g., SMOTE), and weighting the loss function have their problems, at least they don't discard precious data.)

REFERENCE

King, Gary, and Langche Zeng. "Logistic regression in rare events data." Political analysis 9.2 (2001): 137-163.

$\endgroup$
0
$\begingroup$

I don't think there is a "more right" approach; there are tradeoffs to both. Undersampling in the second approach risks throwing away useful information about the relationship between the response and regressors in the majority class. Cost-sensitive approaches risk overfitting to the relationships observed in the minority class.

So yes, see how it turns out in your application. Optimize both and then compare the cv accuracies. Some more advanced alternatives also exist, if you want to give them a go. For instance, SMOTE might be considered a more advanced/better alternative to simply undersampling/oversampling.

$\endgroup$
0
$\begingroup$

They are both correct. There's no conflict between these approaches because there are different upweights: sample weights and class weights.

When you adopt the downsample-upweight approach, you downsample the majority class, and upweight the sample weights of the same class, this means you are using fewer samples but each of the samples has higher weights, this is intuitive.

When you adopt the upweight the minority approach, you are upweighting the class weights to tell the model to treat this class as more important.

Hopefully this helps.

$\endgroup$
2
  • 1
    $\begingroup$ Welcome to Cross Validated! While both approaches are common, they make little sense from the standpoint of the statistics on which machine learning is based. If you read the comments Tim and I wrote to the original question, there is discussion about why and links to blog posts about this subject by Frank Harrell, founding chariman of the Department of Biostatistics and Vanderbilt University Medical Center. $\endgroup$
    – Dave
    Commented Mar 11, 2023 at 19:18
  • $\begingroup$ Hi Dave, thanks for commenting. I understand your point that using proper statistical metrics will largely solve model training issues (for a given data set). My point is on the other side of the problem, where a training data set is not representative of the true distribution of the problem due to extreme class imbalance. In some cases, the imbalance is so extreme that is unrealistic to train on the true distribution, and the model will fail on inference. The down-sampling is a good practice to re-sample the dataset for training. I see this as a data-engineering approach rather than modelling $\endgroup$
    – Y YANG
    Commented Mar 19, 2023 at 16:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.