I have a binary classification problem where my data is highly imbalanced (80:20). Given this imbalance is present in both training and test set, does it make sense to apply specific strategies during training to handle the imbalance problem?
A few general strategies:
- First and foremost, in imbalanced classification problems you want to do stratified cross-validation. This allows you to train your models with the same distribution in your samples.
- Second, you should probably use Cohen's Kappa metric when tuning your models. It is better in imbalanced scenarios because it takes into account random chance as well. A more detailed description was provided in the answer to this question
- If you are adventurous, you can look into cost-sensitive machine learning. In these methods you essentially tell the algorithm that it is better to positively identify certain classes. For example, it would be much worse to misidentify a person with cancer as opposed to accurately identifying them. There many methods including sampling (over, under, SMOTE, SMOTEBoost and EasyEnsemble as referenced in this prior question regarding imbalanced datasets and CSL), Weighting, Thresholding, and Ensemble Methods. These are mostly algorithm agnostic methods, there are also algorithms with CSL builtin but I think this is enough to get your started.