I am taking my first steps in machine learning and data science area. I know for sure that my next task will be related to the imbalanced class problem. I’ve walked through many articles covering this topic, but still have some concerns I hope you can help me with. It's like doing puzzles - you are almost there, but something is missing. And you realize one element was eaten by your dog...
I’ve listed most common ways to overcome the imbalanced class issue, providing some of my comments. Please verify them.
• Change the performance metric. That’s pretty clear to me. The standard classification metric – accuracy – will be misleading, because even the most naive model – always predicting the most common class (let’s say “Yes”) - will have very high accuracy (i. e. 90% if the ratio between two classes equals 9:1). Depending on the case, recall/precision/f1 score are much more reliable. But does it mean I can perform any model on a imbalanced data set and just use some more reliable metrics to verify the results? I thought it can affect a model performance, so it predicts the most common class more willingly and thus, the metrics, no matter which ones, will be – let me say – biased at some way.
Def naive_model(): return “Yes”
• Oversampling minority/Undersampling majority class techniques. Let’s assume I’ve used one of them, without discussing which is worse and which better (it’s arguable and not important right now). I’ve used it AFTER splitting into train and test sets, and while performing cross_val_score, a Pipeline object was used to avoid data leakage during the cross validation. And here is my question. Once the classes are balanced now, does it mean that the accuracy becomes useful?
• Penalize Algorithms. The same scenario as above. I’ve used a penalized algorithm, increasing the cost of classification mistakes on the minority class. How about using the accuracy as a performance metric right now?
• Use tree ensembles algorithms. Literature says they perform well on imbalanced classes. But will they perform as good as if the classes were balanced?
• Can we imagine a scenario, when it’s useful to use more than one technique at a time? Also, how about using a stratified cross validation in cross_val_score? Will it make any sense?
Thank you guys in advance. Really appreciate your time.