I'm currently trying to analyze Tweets and classify them as either positive, negative, or neutral using the NLTK library in Python.
I can see that there's potential in the approach that I'm taking, however, I'm having trouble with my feature selection process.
Indeed, input from Twitter isn't exactly conventional, so there aren't many examples of using TF-IDF to remove uninteresting words in Tweets.
My implementation of the algorithm is working alright, but I don't now how many words I should be cutting out of my feature list based on the TF-IDF scores: 25%? 15%? 30%?
Additionally, and this is the main problem, since Tweets are different than normal English, many uninteresting words, such as 'the' and 'is', don't occur that often, and are therefore assigned good scores by the TF-IDF algorithm.
Now, I know I can use a stoplist, but if I could avoid it, I'd be happy (I have a multitude of reasons not to use a stoplist).
So, to summarize, is there any way to circumvent this issue of uninteresting words getting good scores with TF-IDF?
Edit: I am using a Naive Bayes classifier.