1
$\begingroup$

I have collected a fairly clean set of data (5,410 documents) to train a text classifier. I am now attempting to improve my classification success. (Note: When I trained/tested the classifier from in-sample data, I scored really well. Now that I'm using out-of-sample data, my success is down about 25%. Bad news, but not unexpected for a total ML newb.)

Does having an equal (or more equal) distribution of training documents improve classification success? For instance, although I have only six categories -- Labels A,B,C,D,E, F -- the number of training docs I have for each label varies widely, as Label A = 328; Label B = 1,973; Label C = 148; Label D = 822; Label E = 40; and Label F = 1,242. Any thoughts?? (Note: I don't actually know if this is a representative distribution of what I'm likely to to see out of sample. Is there an ideal algorithm to help with this uncertainty?)

Presently I am using Scikit's svm.LinearSVC() to classify. Thank you for any insight!

$\endgroup$
1
$\begingroup$

Yes, it can affect the classification in a dramatic way. You can use this parameter from the scikit-learn LinearSVC documentation here to compensate the imbalance.

class_weight : {dict, ‘balanced’}, optional

Set the parameter C of class i to class_weight[i]*C for SVC. If not given, all classes are supposed to have weight one. The “balanced” mode uses the values of y to automatically adjust weights inversely proportional to class frequencies in the input data as n_samples / (n_classes * np.bincount(y))

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.