5
$\begingroup$

As far as I can tell, broadly speaking, there are three ways of dealing with binary imbalanced datasets:

Option 1:

Option 2:

  • Create k-fold Cross-Validation samples randomly (or even better create k-fold samples using Stratified k-fold).
  • Do not apply any resampling technique.
  • Use an "alternative" metric for evaluation: for instance the AUC of the Precision-Recall curve or something like the F-score (the harmonic mean of Precision and Recall).

Option 3:

  • Use something like XGBoost and tune the scale_pos_weight ( https://xgboost.readthedocs.io/en/latest/tutorials/param_tuning.html ).
  • Create k-fold Cross-Validation samples randomly (or even better create k-fold samples using Stratified k-fold).
  • Use a "traditional" metric for evaluation: for instance the AUC of the ROC curve (TP Rate vs FP Rate).

My main question is if I correctly interpret what the options are. Is there any conceptual mistake in what I'm saying? Is it appropriate to use Stratified k-fold in the three cases when dealing with imbalance? Is it not necessary to apply any resampling when using XGBoost and tuning scale_pos_weight? When some resampling is applied (Options 1 and 3) does it make sense to use a "traditional" metric and does not make sense to use an "alternative" metric? In general, the resampling has to be applied separately on training and test sets? Etc.

Also, it would be nice if you have any good reference to SMOTE and ROSE, regarding how they work, how to apply them and how to use them with python.

$\endgroup$

1 Answer 1

4
$\begingroup$

Unbalanced datasets are not a problem, and oversampling will not solve a non-problem. Please note Matthew Drury's highly upvoted comment at that thread:

Honestly, knowing there is someone else out there that is mystified by the endless class balancing questions is comforting.

Much of the purported problems probably stems from using accuracy as a KPI, which is a bad idea.

$\endgroup$
4
  • $\begingroup$ I think the discussions you are pointing to are very helpful. And I agree that using accuracy would be wrong. However, I found this post: towardsdatascience.com/… that seems to explain pretty well how one can re-work the problem by using cost-based classification or (almost equivalently) classes re-weight, to take into account the different cost of FP and FN we might have in a specific use case. I suppose that is not in contrast with the mentioned discussions. $\endgroup$
    – Newbie
    Jun 8, 2020 at 16:37
  • $\begingroup$ Now, my remaining questions are limited to: 1) whether is it's a good idea to always apply stratified k-fold, even in the case when there is no class imbalance and 2) if the re-sampling has to be applied only to the k-fold test sets and not to the k-fold training sets. I think I know the answers, but I would be happy to get somebody give a suggestion. $\endgroup$
    – Newbie
    Jun 8, 2020 at 16:37
  • $\begingroup$ I have argued elsewhere that conflating the classification and the decision aspect makes for an intransparent process. (For one, you may have more than two decisions, even if you have two classes: "treat as A" vs. "treat as B" vs. "unsure, get more data/run more tests".) And I don't have much experience in stratified resampling, sorry. It may be a good idea if you have very little data. I would suggest to run your model with and without it and see which one gives better probabilistic predictions, with evaluation based on proper scoring rules. $\endgroup$ Jun 9, 2020 at 5:25
  • $\begingroup$ @StephanKolassa Oversampling will not solve a non-problem, but undersampling might solve an actual problem: high computational effort if you need huge amounts of majority class samples to achieve a realistic class ratio. Undersampling the majority class is very desirable in many domains for this reason, e.g., in image analysis. Then the question remains how to correct the obtained probability estimates for this artificially induced dataset shift. $\endgroup$
    – Eike P.
    Mar 10, 2022 at 20:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.