How to deal with low frequency examples in classification?

Question

I'm facing a text classification problem, and I need to classify examples to 34 groups.

The problem is, the size of training data of 34 groups are not balanced. For some groups I have 2000+ examples, while for some I only have 100+ examples.

For some small groups, the classification accuracy is quite high. I guess those groups may have specific key words to recognize and classify. While for some, the accuracy is low, and the prediction always goes to large groups.

I want to know how to deal with the "low frequency example problem". Would simply copy and duplicate the small group data work? Or I need to choose the training data and expand and balance the data size? Any suggestions?

Answer 1

For sake of simplicity let's assume you're doing binary classification, everything I'll say generalizes straightforwardly to the multiclass case, with $D = X \times Y$ your dataset and $P(Y=1) < P(Y=0)$. This can be a serious problem in real world scenarios since in problems with unbalanced datasets (as above), one is typically much more interested in the under represented class, or at least would prefer to predict each class equally well.

There are several techniques that are typically employed in Machine Learning to remedy the situation.

1. Oversampling: sample a new dataset $D^\prime$ from $D$ such that $P(Y=1) \approx P(Y=0)$ in the new dataset and $|D^\prime| > |D|$. In this scenario you end up with many duplicates of the positive class.

2. Undersampling: sample a new dataset $D^\prime$ from $D$ such that $P(Y=1) \approx P(Y=0)$ in the new dataset and $|D^\prime| < |D|$. In this scenario you end up leaving out some examples from the negative class.

3. Cost Sensitive Learning: here we associate a different cost $C(\text{actual}, \text{predicted})$, (typically a scalar value in $[0,1]$) with different types of missclassifications. To deal with class imbalance in this framework we may say the cost of miss-classifying the minority class is greater than the cost miss-classifying the majority class. Sticking with our example we would have $C(1, 0) > C(0, 1)$. The nice thing about this from a practical perspective is if you are optimizing some per element loss function, this essentially amounts to multiplying that per element loss function by a scalar value, which isn't going to complicate things much from an optimization perspective.

4. Fixing things after the learning: When doing binary classification typically one uses a decision rule like classify $x$ as 1 if $P(y=1|x)/P(y=0|x) > b$, with $b = 1$. However by adjusting $b$ you can make the classifier "prefer" different types of classifications after learning. A common way to choose a good value for $b$ is to look at precision/recall, f-score, AUC, etc, curves on a heldout validation set. I'm not sure if this technique has a proper name, but it is the approach I've had the most success with in applied setting.

For a more in depth look at these techniques see the paper The class imbalance problem in pattern classification and learning.

Note: I've copied this answer from one I gave for a similar question on the soon to be defunct ML stackexchange. I don't know if this is bad form of not.