I have a rather large collection of text documents categorized into about 150 categories. While some categories are represented by several thousands of documents, others have only a few hundreds assigned to them. Now I would like to construct a balanced corpus from this data, where each category is represented by the same number of documents and which at the same time maximizes the categorization accuracy. I tried to randomly select documents from the stronger categories but would like to know if there is a more systematic way.
Choose the minimum class size, M, select the M nearest instances to the class means.
Find the maximum class size, N. Add Gaussian random noise to a random sample with replacement of N-Si instances for each class, other than the largest. S is the class size for the ith class.
For each class, except the smallest, run k-means clustering. k is the number of instances in the smallest class. Use the cluster centers as the instances for the class.
It may be worth mentioning that with option 2 you want to take care not to let the generated additive nose samples leak into the test set for your evaluation.