I'm facing a scenario with 5 classes where a tabulation of the target variable yields:
> 1 2 3 4 5 > 1010 1310 1080 2700 2620
As you can see, classes 4 and 5 are slightly oversampled. If I ignore class imbalance and train a model on all 5 classes, prediction on the test set will only detect classes 4 and 5 and will completely miss the rest (used Random Forest and multinomial logit).
First, I undersampled classes 2, 4, and 5 to 1000 input samples and trained with the same algorithms. Things improved since 1, 2, 3 were being detected now, but individual prediction and recall was relatively low for all classes at around 20%-30% (with accuracy on classes 4 and 5 being worse than before). Finally, without using any resampling for class balance, I split the data into two separate subsets, one for classes 1, 2, 3 and one for classes 4, 5, then trained two models - one for every subset. The results improved radically as precision and recall almost doubled! This is clearly the best path to choose but I'm unable to explain why this is.
Basically I'm not able to justify that, when I use separate models, everything suddenly improves. It's almost as if there are two unrelated datasets (classes 1-3 vs 4-5) that create noise to each other when they are merged. How can this phenomenon be interpreted? What insights does this give me on my data? Is there a better solution to try out instead of training two separate models? A more specialised algorithm perhaps? (I've yet to try XGBoost).