I have a rather large collection of text documents categorized into about 150 categories. While some categories are represented by several thousands of documents, others have only a few hundreds assigned to them. Now I would like to construct a balanced corpus from this data, where each category is represented by the same number of documents and which at the same time maximizes the categorization accuracy. I tried to randomly select documents from the stronger categories but would like to know if there is a more systematic way.
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
3
Some suggestions:
Choose the minimum class size, M, select the M nearest instances to the class means.
Find the maximum class size, N. Add Gaussian random noise to a random sample with replacement of N-Si instances for each class, other than the largest. S is the class size for the ith class.
For each class, except the smallest, run k-means clustering. k is the number of instances in the smallest class. Use the cluster centers as the instances for the class.
It may be worth mentioning that with option 2 you want to take care not to let the generated additive nose samples leak into the test set for your evaluation.
-
$\begingroup$ Thanks four your suggestions. I tried number one and constructed a corpus by selecting the 100 documents with the smallest Euclidean distances from the class centroids (122 classes and 12200 documents in total). The categorization performance in terms of average F1 was 0.485 compared to 0.319 in the case of randomly selected documents. Next, I am going to try the other two. $\endgroup$ Commented Jul 5, 2012 at 6:34
-
$\begingroup$ @mathias I doubt if these approaches are optimal, but at least things are going in the right direction :) $\endgroup$ Commented Jul 5, 2012 at 6:55
-
$\begingroup$ @mathias Let us know how it goes, cheers :) $\endgroup$ Commented Jul 5, 2012 at 10:11