I'm facing a scenario with 5 classes where a tabulation of the target variable yields:

>    1    2    3    4    5  
> 1010 1310 1080 2700 2620

As you can see, classes 4 and 5 are slightly oversampled. If I ignore class imbalance and train a model on all 5 classes, prediction on the test set will only detect classes 4 and 5 and will completely miss the rest (used Random Forest and multinomial logit).

First, I undersampled classes 2, 4, and 5 to 1000 input samples and trained with the same algorithms. Things improved since 1, 2, 3 were being detected now, but individual prediction and recall was relatively low for all classes at around 20%-30% (with accuracy on classes 4 and 5 being worse than before). Finally, without using any resampling for class balance, I split the data into two separate subsets, one for classes 1, 2, 3 and one for classes 4, 5, then trained two models - one for every subset. The results improved radically as precision and recall almost doubled! This is clearly the best path to choose but I'm unable to explain why this is.

Basically I'm not able to justify that, when I use separate models, everything suddenly improves. It's almost as if there are two unrelated datasets (classes 1-3 vs 4-5) that create noise to each other when they are merged. How can this phenomenon be interpreted? What insights does this give me on my data? Is there a better solution to try out instead of training two separate models? A more specialised algorithm perhaps? (I've yet to try XGBoost).

  • $\begingroup$ How would you then use these models to make new predictions? $\endgroup$ – user2974951 Jan 28 '19 at 8:39
  • $\begingroup$ @user2974951 I can't! That's the problem. That's why this solution fails. I tried to create a pipeline with a third classifier that would separate classes {1,2,3} from {4,5} but that didn't work so I'm all out of ideas. $\endgroup$ – en1 Jan 28 '19 at 10:16

When you partition your data based on the information available in the training set you are already doing a sort of classification before the actual model classification. You are in effect "helping" your models perform better by manually doing some of the work. In doing so you can actually make pretty good models, but this won't generalize well, since you won't have such information available in the new data (and probably also overfitting), which renders this approach quite useless. In summary, you have to only use information that will be available to the model in the future as well, that is the model has to learn by itself.

Some other options would be to 1) upsample the smaller classes via SMOTE, 2) aggregate some of the classes if they are quite similar, 3) use a different model.

  • $\begingroup$ For me class rebalancing only makes sense when the class distribution of the population is known and different to class distribution in the training sample. What I mean is, if you train a model on a rebalanced sample and then try to make a prediction on unseen data where classes are as unbalanced as they were in your original training set - then the prediction will be trash. $\endgroup$ – en1 Jan 28 '19 at 19:02
  • $\begingroup$ I was wondering whether it would make sense to use a pipeline of models here, i.e. first a binary classifier to automate the process of the manual work that I'm doing, then the two individual classifiers that I mentioned earlier. Would something like that be correct? $\endgroup$ – en1 Jan 28 '19 at 19:04
  • $\begingroup$ @en1 I don't know how such a process would work, but if you can make it work then sure, that is if you can achieve a high score on your test set. Although this is unconventional and you should try the different approaches I mentioned first, before you reinvent the wheel. $\endgroup$ – user2974951 Jan 30 '19 at 11:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.