Imbalanced classess and how to deal with them

Question

I am taking my first steps in machine learning and data science area. I know for sure that my next task will be related to the imbalanced class problem. I’ve walked through many articles covering this topic, but still have some concerns I hope you can help me with. It's like doing puzzles - you are almost there, but something is missing. And you realize one element was eaten by your dog...

I’ve listed most common ways to overcome the imbalanced class issue, providing some of my comments. Please verify them.

• Change the performance metric. That’s pretty clear to me. The standard classification metric – accuracy – will be misleading, because even the most naive model – always predicting the most common class (let’s say “Yes”) - will have very high accuracy (i. e. 90% if the ratio between two classes equals 9:1). Depending on the case, recall/precision/f1 score are much more reliable. But does it mean I can perform any model on a imbalanced data set and just use some more reliable metrics to verify the results? I thought it can affect a model performance, so it predicts the most common class more willingly and thus, the metrics, no matter which ones, will be – let me say – biased at some way.

Def naive_model():
    return “Yes”

• Oversampling minority/Undersampling majority class techniques. Let’s assume I’ve used one of them, without discussing which is worse and which better (it’s arguable and not important right now). I’ve used it AFTER splitting into train and test sets, and while performing cross_val_score, a Pipeline object was used to avoid data leakage during the cross validation. And here is my question. Once the classes are balanced now, does it mean that the accuracy becomes useful?

• Penalize Algorithms. The same scenario as above. I’ve used a penalized algorithm, increasing the cost of classification mistakes on the minority class. How about using the accuracy as a performance metric right now?

• Use tree ensembles algorithms. Literature says they perform well on imbalanced classes. But will they perform as good as if the classes were balanced?

• Can we imagine a scenario, when it’s useful to use more than one technique at a time? Also, how about using a stratified cross validation in cross_val_score? Will it make any sense?

Thank you guys in advance. Really appreciate your time.

Answer 1

Many models output something other than a hard classification. While particular software implementations might make it easy or difficult to access these, you get the full probability of class membership from models such as logistic regressions, neural networks, and random forests.

I would argue that, if you do not know the cost of making a mistake, you have no business making a hard classification, and all you should be doing is estimating the probability.

Consequently, the goal should be to obtain the best probability predictions you can. Two typical metrics for this go by the names of log-loss and Brier score. Both of these are perfectly compatible with class imbalance, as they will (correctly) use the imbalance to inform their probability predictions. In other words, you need overwhelming evidence of a minority class, because the majority class is so much more likely.

In technical terminology, the posterior probability $P(\textrm{category}\vert \textrm{data})$ depends on the prior probability $P(\textrm{category}).$ A machine learning model that outputs a probability value is predicting the posterior probability. Using Bayes' theorem, we can see that this value depends on the prior probability (the class membership proportion)

$$ P(\textrm{category}\vert \textrm{data}) = \frac{ P(\textrm{data}\vert \textrm{category}) P(\textrm{category}) }{ P(\textrm{data}) }. $$