I'm working on a problem to predict/classify overall sentiment of a large amount of text, which I can verify on the next day. Each data point is a day and is composed of multiple articles. I bin the words by frequency in those articles to train a Bayesian classifier.

I tried predicting the sentiment using all previous data. So on day N I would use all data from 1:N-1. However the accuracy of prediction over 110 test examples was only 51%.

I noticed that the predictions were getting worse overtime, because the first 20 test examples were accurate at 70%. When i used only 1:20 examples in my training set for all subsequent predictions, the model was accurate at a rate of 57%.

So my question is... How do determine the best features in that initial 20 days worth of data. Should I just use the top X features? Or is this 57% just randomness tempting me to follow an unfruitful path? Might there be a better model to use to classify binary sentiment of a large amount of text as positive or negative (multinomial bayes, sum)?

Thanks, Ricky

  • $\begingroup$ I think the answer to all your questions is yes possibly. To have a better idea would require knowing more details about your data. $\endgroup$ – Michael R. Chernick Sep 16 '12 at 21:13
  • 1
    $\begingroup$ Perhaps try using the days (N-20):(N-1) of data? If the data is moving out from under you, that could cause what you've seen. $\endgroup$ – John Salvatier Sep 17 '12 at 17:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.