How to deal with that accuracy of minority class improves while majority class decreases when using imbalance learning?

Question

I use boosting tree to make prediction for the stock direction, and it is a binary class classification.

The majority class is the down direction, and the minority class is the up direction. The boosting tree can make a better prediction for the majority class compared with minority class, or you can regard that the tree somewhat overfits to the majority class.

So I leave the test set as the same, while impose a bigger weight for the training loss calculation in the minority class of the training set. Finally, I observe that the whole loss of the test set can increase only a bit or decrease a lot, which depends on the weight imposed in the training loss. BUT, the worst thing is that the accuray of the majority class always decreases.

My question is that how can I improve the accuracy for both the majority class and minority class when using imbalance learning techs (some other imbalance learning methods can also be considered), Is it possible?

Answer 1

When you weight the loss function like you have, you are telling the model to prefer making predictions that favor the minority class over the majority. This means that you are telling the model to sacrifice detection of the majority class in favor of detecting the minority class. It is only natural that you would have your performance in detecting the majority class suffer. You might be willing to accept such a tradeoff, but do be aware that such a tradeoff is natural. In the extreme, you can miss absolutely zero instances of membership in the minority category by simply predicting every instance to be in the minority category.

By weighting the loss function, oversampling the minority class (e.g., SMOTE, ROSE), or undersampling the majority class, you are not really introducing new information that helps the model discriminate between categories. If you want to get better predictions of both the majority and minority classes, you need to get data that is better-able to discriminate between your categories (e.g., measure a new variable that has an influence on the outcome) or use your existing data in a better way (e.g., better feature extraction/engineering, using flexible models). As always, watch out for overfitting.

Also note the considerable issues with using accuracy and related notions. I will leave some of the usual links I post on this subject, which often arises in the context of class imbalance but makes sense for balanced classes, too.

Are unbalanced datasets problematic, and (how) does oversampling (purport to) help?

Profusion of threads on imbalanced data - can we merge/deem canonical any?

Why is accuracy not the best measure for assessing classification models?

Academic reference on the drawbacks of accuracy, F1 score, sensitivity and/or specificity

Finally, it is not obvious that predicting the direction of a price movement is particularly useful. Yes, buying before gains and selling before losses sounds like a good strategy, but it is possible to have trading fees and taxes eat into your profits, perhaps so much that you wind up in the red. Further, you can make the right decision most of the time yet lose money because your misclassifications occur when the magnitudes are large, wiping out the gains you have from frequently predicting the right direction when the magnitudes are small (which might not even wind up profitable because of trading fees and taxes, anyway).

Answer 2

Don't use accuracy as a KPI. Why is accuracy not the best measure for assessing classification models? Instead, use probabilistic class membership predictions and assess these using proper scoring rules.

What you are doing is related to over-/undersampling. It's better to use proper scoring rules. Are unbalanced datasets problematic, and (how) does oversampling (purport to) help?