Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 5 down vote favorite
5

I'm currently trying to analyze Tweets and classify them as either positive, negative, or neutral using the NLTK library in Python.

I can see that there's potential in the approach that I'm taking, however, I'm having trouble with my feature selection process.

Indeed, input from Twitter isn't exactly conventional, so there aren't many examples of using TF-IDF to remove uninteresting words in Tweets.

My implementation of the algorithm is working alright, but I don't now how many words I should be cutting out of my feature list based on the TF-IDF scores: 25%? 15%? 30%?

Additionally, and this is the main problem, since Tweets are different than normal English, many uninteresting words, such as 'the' and 'is', don't occur that often, and are therefore assigned good scores by the TF-IDF algorithm.

Now, I know I can use a stoplist, but if I could avoid it, I'd be happy (I have a multitude of reasons not to use a stoplist).

So, to summarize, is there any way to circumvent this issue of uninteresting words getting good scores with TF-IDF?

Edit: I am using a Naive Bayes classifier.

share|improve this question
1  
Words uninteresting in normal English (articles, etc.) may actually be relevant in tweets: that would be the case, for instance, if you change the style in which you write depending on the positive or negative nature of the message. (That may be one of your reasons not to use a stoplist.) – Vincent Zoonekynd Jan 15 '12 at 13:07
Ah, I understand. So it is normal that when a tweet contains the word 'is', it has a 8 to 1 chance of being negative? – boopyman Jan 15 '12 at 14:50
What classification method are you using? – tdc Jan 17 '12 at 16:16
I am using a Naive Bayes classifier. – boopyman Jan 17 '12 at 21:20

1 Answer

up vote 2 down vote accepted

Probably the most effective (but also timeconsuming) approach will be to hand pick a set of examples that you know are postive, negative, and neutral. You can then train a classifier (Naive Bayes, SVM, Fisher Discriminant or whatever) on these examples (since you are using 3 classes, you will need to do multi-class classification, although to begin with it might simplify your problem to only look at +ve/-ve and introduce the neutral class later). You should ensure that you have enough examples so that you can perform k-fold cross-validation of the classifier hyperparameters effetively. The more training examples you have, the better the estimation of your threshold will be.

Without any training examples, you will have to resort to ad-hoc rules which are unlikely to be robust.

share|improve this answer
I'm already using a Naive Bayes classifier, but by now, I've dropped the idea of using TF-IDF. I will try your suggestion and attempt k-fold-cross-validation on more training examples ! – boopyman Feb 3 '12 at 22:16
2  
@boopyman by the way it's a common misconception that Naive Bayes is somehow the "right" classifier for text data. It just so happens that one of the first papers to attempt it (SPAM email detection) used Naive Bayes and it worked well, but actually the task was quite easy and pretty much any classifier would have worked well. If you're planning on using large datasets (millions of examples) I would look at classifiers that scale better - e.g. variants of the SVM such as pegasos or liblinear – tdc Feb 4 '12 at 9:26
Thanks for the suggestion tdc ! – boopyman Feb 4 '12 at 13:37

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.