I have collected a fairly clean set of data (5,410 documents) to train a text classifier. I am now attempting to improve my classification success. (Note: When I trained/tested the classifier from in-sample data, I scored really well. Now that I'm using out-of-sample data, my success is down about 25%. Bad news, but not unexpected for a total ML newb.)
Does having an equal (or more equal) distribution of training documents improve classification success? For instance, although I have only six categories -- Labels A,B,C,D,E, F -- the number of training docs I have for each label varies widely, as Label A = 328; Label B = 1,973; Label C = 148; Label D = 822; Label E = 40; and Label F = 1,242. Any thoughts?? (Note: I don't actually know if this is a representative distribution of what I'm likely to to see out of sample. Is there an ideal algorithm to help with this uncertainty?)
Presently I am using Scikit's
svm.LinearSVC() to classify. Thank you for any insight!