How Does a Disparity in Number of Documents (Training Data Points) Affect Text Classification?

Question

I have collected a fairly clean set of data (5,410 documents) to train a text classifier. I am now attempting to improve my classification success. (Note: When I trained/tested the classifier from in-sample data, I scored really well. Now that I'm using out-of-sample data, my success is down about 25%. Bad news, but not unexpected for a total ML newb.)

Does having an equal (or more equal) distribution of training documents improve classification success? For instance, although I have only six categories -- Labels A,B,C,D,E, F -- the number of training docs I have for each label varies widely, as Label A = 328; Label B = 1,973; Label C = 148; Label D = 822; Label E = 40; and Label F = 1,242. Any thoughts?? (Note: I don't actually know if this is a representative distribution of what I'm likely to to see out of sample. Is there an ideal algorithm to help with this uncertainty?)

Presently I am using Scikit's svm.LinearSVC() to classify. Thank you for any insight!

Answer 1

Yes, it can affect the classification in a dramatic way. You can use this parameter from the scikit-learn LinearSVC documentation here to compensate the imbalance.

class_weight : {dict, ‘balanced’}, optional

Set the parameter C of class i to class_weight[i]*C for SVC. If not given, all classes are supposed to have weight one. The “balanced” mode uses the values of y to automatically adjust weights inversely proportional to class frequencies in the input data as n_samples / (n_classes * np.bincount(y))