How to construct an optimally balanced reference corpus for text categorization?

Question

I have a rather large collection of text documents categorized into about 150 categories. While some categories are represented by several thousands of documents, others have only a few hundreds assigned to them. Now I would like to construct a balanced corpus from this data, where each category is represented by the same number of documents and which at the same time maximizes the categorization accuracy. I tried to randomly select documents from the stronger categories but would like to know if there is a more systematic way.

Answer 1

Some suggestions:

1. Choose the minimum class size, M, select the M nearest instances to the class means.

2. Find the maximum class size, N. Add Gaussian random noise to a random sample with replacement of N-Si instances for each class, other than the largest. S is the class size for the ith class.

3. For each class, except the smallest, run k-means clustering. k is the number of instances in the smallest class. Use the cluster centers as the instances for the class.

It may be worth mentioning that with option 2 you want to take care not to let the generated additive nose samples leak into the test set for your evaluation.