I have a text classification problem to solve. I need to classify a given sample of text into one of two classes A or B.

My training set has about 30% A and 70% B. This is my prior. Now, when I build the classifier, I get a confusion matrix like so

        A    0.65    0.35 
Class   B    0.20    0.80

             A         B
           Predicted class

One issue that I noticed is that when most of the features are zero, the training data classifies the sample as A. However, the classifier has a prior bias towards B so if I get an evaluation sample that has very few features, I will misclassify it as A.

I attempted to remedy this by biassing the classifier towards A. When I do this, My matrix improves. The False negatives and false positives both reduce to 0.18 which is an improvement.

What I want to know is whether this makes sense from a mathematical perspective.

  • $\begingroup$ Before I can comment further, what are your features and which classifier are you using? $\endgroup$ – Jaidev Deshpande Dec 23 '18 at 7:10
  • $\begingroup$ I've tried word ngrams and letter ngrams. We evaluated a few classifiers but finally settled on binomial logistic. $\endgroup$ – Noufal Ibrahim Dec 23 '18 at 10:22
  • $\begingroup$ Thanks. Here's my analysis, let me know if it makes sense: Since you've used ngrams, your feature matrix is probably very sparse. So no matter what the class, most of your features would be zeros anyway. So when you say that the classifier tends to think of a class containing more zeros as A, I think it means that class B simply contains a wider variety of words. Does this make sense? Getting to your question - how did you bias the classifier towards A? Did you flip the distribution in favour of the underrepresented class and use this flipped distribution as a prior? $\endgroup$ – Jaidev Deshpande Dec 24 '18 at 15:06
  • $\begingroup$ You're right on both questions. $\endgroup$ – Noufal Ibrahim Dec 24 '18 at 16:46
  • $\begingroup$ Cool. This sounds perfectly valid to me. This is done all the time for imbalanced classes. Check the class_weight parameter of scikit-learn.org/stable/modules/generated/… $\endgroup$ – Jaidev Deshpande Dec 24 '18 at 16:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.