2
$\begingroup$

I have a database of data that is labelled good. I can make new data myself and I'll label it bad (I know how to transform a bad data from a good data).

I want my model to determine if a piece of data is good (not modified) or bad (modified), I wanted to do 50/50 in my training dataset but I know that in real life, bad data only occurs like around 0.1% of the time.

Should I reproduce this in my training dataset? Why? Any intuition about why?

$\endgroup$
4
  • $\begingroup$ Why do you want 50/50 in the training data when you know that to be a wildly incorrect class ratio? $\endgroup$
    – Dave
    Nov 29, 2022 at 13:55
  • $\begingroup$ Because I've read that using 0.01/99.9 would lead to the algorithm always returning the class of the majority label. I'm also looking for an intuition as to why a wildly incorrect class ratio would be bad $\endgroup$ Nov 29, 2022 at 13:57
  • $\begingroup$ Class imbalance is such a common source of confusion that we have an entire Meta question devoted to answering it. Pay particular attention to the Stephan Kolassa link. $\endgroup$
    – Dave
    Nov 29, 2022 at 14:01
  • $\begingroup$ Thanks! I'm gonna look at it, hopefully I'll understand why doing 50/50 is bad if it's not the case in real life! $\endgroup$ Nov 29, 2022 at 14:17

1 Answer 1

2
$\begingroup$

Because I've read that using 0.01/99.9 would lead to the algorithm always returning the class of the majority label.

Most models, such as logistic regressions, neural networks, and random forests, give predictions on a continuum, rather than making hard classifications. The way software packages get a categorical prediction is by comparing that continuous prediction to some kind of threshold and deciding on the category by seeing the side of the threshold on which the prediction falls (above the prediction goes to the positive category, below to the negative). These continuous predictions often lie in the interval $[0,1]$ and can have an interpretation as probability, so the typical threshold is $0.5$.

In an unbalanced problem, it might be that the probability of being in the minority class never exceeds $0.5$, meaning that correct performance might be to having every prediction fall below this standard threshold. Consequently, some suggest to artificially balance the categories so the predictions can be higher. After all, a higher prior probability (the proportion of observations that belong to the minority class) leads to a higher posterior probability by Bayes' theorem, all else equal. Below, $P(Y=1)$ is taken as the prior probability.

$$ P(Y = 1\vert X = x) = \dfrac{ P(X = x\vert Y = 1)P(Y = 1) }{ P(X = x) } $$

Not all else is equal, but this is the idea behind artificial balancing (e.g., ROSE, SMOTE), and the results show that artificial balancing indeed increases predicted probabilities.

Consequently, this artificial balancing can be expected to be successful in making the model make higher predictions that exceed the $0.5$ threshold more often. However, your predictions are now, in some sense, not telling the truth. A prediction of $0.8$ does not mean the event is expected to happen with probability $0.8$. Consequently, the rich information present in probabilistic predictions from your model no longer tell the truth. While you might be able to calibrate these predictions to reflect the reality of how often events occur, fiddling with the class ratio and then calibrating (maybe) just to make sure uncalibrated predictions fall above an arbitrary threshold seems inefficient at best (there could be edge cases where this winds up working better, sure) and damaging to the education of aspiring predictive modelers at worst by eliminating an emphasis on the valuable information available in the original, non-categorical predictions.

If you start out with the natural class ratio, you do not start the modeling by misleading the model about the prior probability. Consequently, most "fixing" of class imbalance is unnecessary (though this question has an answer with an interesting edge case) and due to an unnecessary obsession with making sure the predictions are on the correct side of $0.5$ when it is fine to use a different threshold (the prior probability seems like a natural one) or consider the direct predictions without considering any threshold at all.

(I know, I know: sklearn uses predict to return categorical predictions. I believe that sklearn, for all of its positives (and I have used it to do good work at my job), has caused damage to the field of statistics by using the predict method to return the category with the highest probability and the predict_proba method to return the raw probabilities, rather than having predict return the probabilities and something like predict_category to return the category with the highest probability.)

I will close with some standard links about class imbalance and evaluating classifiers.

Are unbalanced datasets problematic, and (how) does oversampling (purport to) help?

Profusion of threads on imbalanced data - can we merge/deem canonical any?

Why is accuracy not the best measure for assessing classification models?

Academic reference on the drawbacks of accuracy, F1 score, sensitivity and/or specificity

$\endgroup$
2
  • $\begingroup$ That's an interesting insight! I'll dive deeper, but from what I understand artificially balancing the datasets lead to probabilities not being interpretable as probabilities anymore. And moving the threshold has the benefit of having probabilities outputted still being interpretable as probabilities? $\endgroup$ Apr 6, 2023 at 9:40
  • $\begingroup$ @FluidMechanicsPotentialFlows There are models that are notorious for returning uncalibrated probabilities whether you artificially balance or not, so it is not quite that simple. However, messing with the data just to adhere to some threshold that might not make any sense for your application is inefficient at best and damaging to the education of aspiring modelers at worst by removing the interest in the useful raw outputs. (Sure, sometimes it helps.) $\endgroup$
    – Dave
    Apr 6, 2023 at 10:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.