Collecting training data for document classification with unbalanced classes

Question

I have a document classification problem in which the estimated class proportions in the population are severely unbalanced: the population is ~99% class 0 and ~1% class 1.

I am using a logistic regression classifier (LibLINEAR), and have little flexibility in this decision.

To maximize the classifier's F1 score, should I try to collect training data with the same class proportions as the population (while ensuring that there are enough instances of the minority class)? Or should I collect equal proportions of the classes, and use asymmetric misclassification penalties?

Answer 1

Using different misclassification penalties is a good idea. The theory which allows you to do this and tells you how to correct for biases is importance sampling.

Say the proportion of class 1 in the real world is $p$, but the proportion in your training set is $q$. Then to correct for this bias, you should weight instances of class 1 by $\frac{p}{q}$ and instances of class 0 by $\frac{1 - p}{1 - q}$. That is, if your cassifier's objective function is

$$\sum_{(x, y) \in \mathcal{P} \cup \mathcal{N}} f(x, y),$$

you need to change it to

$$\sum_{(x, y) \in \mathcal{P}} \frac{p}{q} f(x_i, y_i) + \sum_{(x, y) \in \mathcal{N}} \frac{1 - p}{1 - q} f(x, y),$$

where $\mathcal{P}$ and $\mathcal{N}$ contain your positive and negative examples, respectively.

Answer 2

I'd recommend collecting documents for a class ratio that reflects what will be observed in the larger population of documents of interest. For example, in biomedical text classification, we often have highly skewed data--positive-class documents tend to be rare--and the classifier needs to be able to deal with this. If you collect an equal proportion of classes for your training data, but the real-world unseen documents you'll be classifying are drawn from an imbalanced distribution, then your performance is going to suffer.