I'm working on a sentiment analysis study of twitter data using the Maximum Entropy classifier. I've gathered dozens of thousands of tweets. To produce features, I used unigram, bigram and dictionary. The first goal is to divide them into topics (also with maxent classifier), and it went well. The second goal is for each topic, assign a sentiment label, either positive (P), neutral (0), or negative (N) to the tweet.

When labeling the training data, I noticed that 'neutral' is assigned to the majority of tweets. Proportionally, I have a P:0:N ratio of 2:10:1. This created a problem that in the model, I have abundant features for 'neutral', but not so many for others. The cross validation result for the model is like 80% precision for all labels, but recall is 40% for N and 50% for P.

Bringing in more training data did not improve recall much.

Generally how do I improve recall on a label that is only 7% of the training data?

  • $\begingroup$ You can try using a balanced sample for training, with similar proportions of each class. Further, you can see how the class distribution of the training sample changes the recall. $\endgroup$ – image_doctor Jan 1 '15 at 11:38
  • $\begingroup$ You mean artificially inject some N label data? Won't this affect the empirical probability distribution and thus affect the entropy principle? $\endgroup$ – He Shiming Jan 1 '15 at 12:56
  • $\begingroup$ You could either inject N and P data or reduce O. Consider a data set with binary labels, {0,1} and a class distribution of 99:1, a classifier which simply predicts 0 for every thing it sees is 99% accurate. What performance do you want, accuracy or specific recall performance. You might want to consider adding a class based cost function for incorrect classification and then optimise for that. You'll probably suffer poorer performance on your O class , but better on N and P. The exact compromise that you feel is best will depend on the performance your application requires. $\endgroup$ – image_doctor Jan 1 '15 at 13:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.