Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I collect a bunch of questions from Twitter's stream by using a regular expression to pick out any tweet that contains a text that starts with a question type: who, what, when, where etc and ends with a question mark.

As such, I end up getting several non-useful questions in my database like: 'who cares?', 'what's this?' etc and some useful ones like: 'How often is there a basketball fight?', 'How much does a polar bear weigh?' etc

However, I am only interested in useful questions.

I have got about 3000 questions, ~2000 of them are not useful, ~1000 of them are useful that I have manually label them. I am attempting to use a naive Bayesian classifier (that comes with NLTK) to try to classify questions automatically so that I don't have to manually pick out the useful questions.

As a start, I tried choosing the first three words of a question as a feature but this doesn't help very much. Out of 100 questions the classifier predicted only around 10%-15% as being correct for useful questions. It also failed to pick out the useful questions from the ones that it predicted not useful.

I have tried other features such as: including all the words, including the length of the questions but the results did not change significantly.

Any suggestions on how I should choose the features or carry on?

Thanks.

share|improve this question
    
What is your definition of a useful question? I am not sure if I can extrapolate your definition by the examples you gave. –  bjoernz Jan 24 '13 at 8:08

2 Answers 2

Interesting problem! Have you tried using other classifiers? I bet you'd get better performance using a support vector machine (SVM) classifier. There's a wealth of literature demonstrating that SVM's can excel in high-dimensional classification problems (such as yours...and many text-classification problems, for that matter). In my experience, Naive Bayes can get mislead by uninformative features, when the problem space gets large.

In terms of other features you could try, it might be interesting to have a feature denoting the co-occurrence of a question-type word (e.g., what, when, how), and a noun (e.g., polar bear or basketball). Have you taken a look at the ones that you've labeled as interesting and considered what the characteristics were that lead you to that conclusion?

share|improve this answer
    
Thanks for the suggestions. I have never worked with SVM before but I wil get started on that. I have tried to look at both sets useful and not_useful questions to see if there is any pattern but it looks very random to me. Im still playing around with different features to see if it can improve the classifier. –  bili Jan 14 '13 at 21:43

With only 3000 examples, your data set is fairly small (tweets are short) for a naive word frequency approach. If you use good priors on word frequencies, you might have better luck. Intuitively, I would expect words with lower frequency in the english language to have a higher rank of "interestingness". There are lots of datasets out there to get these word frequencies from.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.