Choosing features to identify Twitter questions as “useful” I collect a bunch of questions from Twitter's stream by using a regular expression to pick out any tweet that contains a text that starts with a question type: who, what, when, where etc and ends with a question mark.
As such, I end up getting several non-useful questions in my database like: 'who cares?', 'what's this?' etc and some useful ones like: 'How often is there a basketball fight?', 'How much does a polar bear weigh?' etc
However, I am only interested in useful questions.
I have got about 3000 questions, ~2000 of them are not useful, ~1000 of them are useful that I have manually label them. I am attempting to use a naive Bayesian classifier (that comes with NLTK) to try to classify questions automatically so that I don't have to manually pick out the useful questions.
As a start, I tried choosing the first three words of a question as a feature but this doesn't help very much. Out of 100 questions the classifier predicted only around 10%-15% as being correct for useful questions. It also failed to pick out the useful questions from the ones that it predicted not useful.
I have tried other features such as: including all the words, including the length of the questions but the results did not change significantly.
Any suggestions on how I should choose the features or carry on?
Thanks.
 A: Interesting problem!
Have you tried using other classifiers? I bet you'd get better performance using a support vector machine (SVM) classifier. There's a wealth of literature demonstrating that SVM's can excel in high-dimensional classification problems (such as yours...and many text-classification problems, for that matter). In my experience, Naive Bayes can get mislead by uninformative features, when the problem space gets large.
In terms of other features you could try, it might be interesting to have a feature denoting the co-occurrence of a question-type word (e.g., what, when, how), and a noun (e.g., polar bear or basketball). Have you taken a look at the ones that you've labeled as interesting and considered what the characteristics were that lead you to that conclusion?
A: With only 3000 examples, your data set is fairly small (tweets are short) for a naive word frequency approach. If you use good priors on word frequencies, you might have better luck. Intuitively, I would expect words with lower frequency in the english language to have a higher rank of "interestingness".  There are lots of datasets out there to get these word frequencies from.
