I am doing a project work in sentiment analysis (on Twitter data) using machine learning approach. In order to find the 'best' way to this I have experimented with naive Bayesian and maximum entropy classifier by using unigrams, bigrams and unigram and bigrams together. I'm using the SharpEntropy library for ME, and an own implementation for the NB.

My question is related to the ME classifier. In one of my tests I've trained it using both unigrams and bigrams because, these outperformed the unigram only and bigram only solutions. I used 100 iterations and no cutoff (= minimum appearance for using it in the training).

After the initial training I've started the validation and the strange thing I encountered is the following: if the test data is not converted to the unigram-bigram representation my classifier was 4% more accurate (73%->77%) than the case when I did the transformation. So basically my classifier is trained using uni- and bigrams, but the test data remained in the unigram representation and I got better results. (unigram training - unigram test data resulted in 58% accuracy, bigram only training - bigram only test resulted in 68% accuray)

What could be the reason for this behavior? I am looking for some help to get a deeper understanding for this.

  • 1
    $\begingroup$ To clarify: Are you using "maximum entropy classifier" as a synonym for logistic regression (via MLE)? $\endgroup$ – cardinal Feb 20 '12 at 17:45
  • 1
    $\begingroup$ Hi cardinal, yes I used it as a synonym. $\endgroup$ – norbip Feb 20 '12 at 18:27
  • 1
    $\begingroup$ I assume that using both feature sets in training but only one in your validation helped preventing overfitting. You could test your current method against one with regularization. $\endgroup$ – Tim Feb 20 '12 at 23:13
  • 1
    $\begingroup$ I agree with Tim as it far as it concerns the logres. However, in NB, shouldn't the independence prevent such an effect ? As long as no parameter optimization is performed in NB, of course. $\endgroup$ – steffen Feb 21 '12 at 14:58
  • $\begingroup$ Thanks for the suggestions. In case of NB yes, the independence assumption prevents this behavior. The increase in accuracy was only observed in case of the ME. (and the reason for seeing this behavior happened because of a small bug in my code :)) $\endgroup$ – norbip Feb 21 '12 at 15:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.