dealing with imbalanced data set in multiclass text classification

Question

I need to build a text classification model.

I have a labeled training set and my goal is to classify the new unlabeled text .

My training set is composed on 6 categories, that are imbalanced.

The categories are distributed as follows: Category 1 -> 450 examples Category 2 -> 400 examples Category 3 -> 250 examples Category 4 -> 150 examples Category 5 -> 100 examples Category 6 -> 50 examples

How to deal with such imbalanced multi class text classification?

Answer 1

Generally, you should:

• Sampling
• Adjust your performance metrics (like F1 rather than just accuracy)
• Choose a cost-sensitive algorithm, for example, adding weights to the minority classes
• Algorithms such as decision tree, boosting etc. They are more adopted to imbalanced data set.

Answer 2

It really depends on your data but there are at least four things you could try:

• Upsample the training set by copying the examples in each category
• Downsample the training set by deleting some examples from the dominating categories
• Use a boosting algorithm like scikit-learn's Adaboost
• Use cost-sensitive classification algorithm

Answer 3

You can downsampling or upsampling but these are vary naive approaches. You could try to intelligently sample the data in order to "re-balance" the dataset. However, in any case: imbalanced data is very tricky and requires a lot of domain knowledge.

• Always measure accuracy with the AUC: This denotes the probability that you can correctly classify a random positive instance and a random negative instance. There is no issue of imbalance here.

• Coevolution with the genetic algorithm works well. You are evolving sub-models to collaborativley work together to solve the given task at hand.

• Bayesian methods also work well here. Provided that you have a good prior distribution

Honestly - that's the best I can think of. Imbalanced classes are lame.