0
$\begingroup$

I'm fairly new to data science.

I have a multi-class classification problem with 4 classes, 100K rows. The problem is that the classes are balanced but the prediction results are not. (All 4 classes have 25K observations)

For example, lets say that the classes are A, B, C, and D. Currently I'm using XGBoost and the model predictions are unbalanced. The number of predicted observations are pretty much like this.

[ A: 5600, B: 3900, C: 5250, D: 5250 ]

I get that the features are not good enough to classify between A and B (Model prediction for class 'C' and 'D' are not bad). But I don't understand why the results are unbalanced. The model tends to predict class B as class A. I've done a lot of CV and parameter optimization but the unbalanced predictions are always there.

Is there any way to improve this unbalanced results? For example, would it make sense to oversample, or use SMOTE to increase class B samples?

Please help.

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ Welcome to our site. I'm not understanding something: how is it possible that the total of the predictions will (greatly) exceed the amount of data? Are you perhaps describing out of sample predictions? If that's so, then could you show us the information you have that allows you to conclude the predictions ought to be balanced? $\endgroup$
    – whuber ♦
    Aug 27, 2018 at 18:24
  • $\begingroup$ Thanks for the comment! It was a typo. There are 100K rows total, 25K for each class! $\endgroup$
    – Jason
    Aug 27, 2018 at 23:15
  • $\begingroup$ The comment still applies: could you tell us what kind of "predictions" these are and explain why their total does not equal the number of rows your dataset and why you expect them to have the same distribution as in the dataset? $\endgroup$
    – whuber ♦
    Aug 28, 2018 at 14:02
  • 1
    $\begingroup$ The examples in the passage are results of cross validation. So distribution should be the same since the folds were selected randomly. $\endgroup$
    – Jason
    Aug 29, 2018 at 8:36
  • $\begingroup$ That may be a crucially important detail, Jason. The specifics of the cross-validation procedure are needed in order to analyze what's going on. Please include that information within your question. $\endgroup$
    – whuber ♦
    Aug 29, 2018 at 14:06

1 Answer 1

Reset to default
0
$\begingroup$

One thing you might try (I am not sure whether this will help though), is to just single out class A and B, and build a binary classifier (with say XGBoost). Since in this case, they are quite balanced, you can probably judge its performance, and maybe draw some insight about your features and hyper-parameters?

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for the reply. I actually tried A, B binary classification but the results were still unbalanced. Found some difference in feature importance, but that was all. Trying different feature subsets didn't really help. $\endgroup$
    – Jason
    Aug 28, 2018 at 13:54
  • $\begingroup$ So it seems that classes A and B are hard to differentiate to start with? As you said the results are unbalanced, it implies that the model (xgboost?) will prefer one over another. This is odd... maybe try some simpler model, say logistics regression with some regularization. The metics (e.g. AUC of ROC) might not be as good as from more complicated models, but you may check how are the 'unbalance' situation to see if there is any difference? $\endgroup$
    – ccy
    Aug 28, 2018 at 13:59
  • $\begingroup$ That seems to be a great idea to start with! Thanks a lot! $\endgroup$
    – Jason
    Aug 29, 2018 at 8:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.