I'm working on a problem to predict/classify overall sentiment of a large amount of text, which I can verify on the next day. Each data point is a day and is composed of multiple articles. I bin the words by frequency in those articles to train a Bayesian classifier.
I tried predicting the sentiment using all previous data. So on day N I would use all data from 1:N-1. However the accuracy of prediction over 110 test examples was only 51%.
I noticed that the predictions were getting worse overtime, because the first 20 test examples were accurate at 70%. When i used only 1:20 examples in my training set for all subsequent predictions, the model was accurate at a rate of 57%.
So my question is... How do determine the best features in that initial 20 days worth of data. Should I just use the top X features? Or is this 57% just randomness tempting me to follow an unfruitful path? Might there be a better model to use to classify binary sentiment of a large amount of text as positive or negative (multinomial bayes, sum)?