Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm looking into a way to classify/categorize sentences into pre-defined categories (around 10-15). Yes, indeed sentences, not articles or paragraphs.

Given the average length of articles are not too long (2-10 normal pages) and number of articles (tens-hundreds) are relatively small, it is quite a small scale problem. However, accuracy is much more important.

Because I am quite new in this field, I started by looking into some introductory papers and few generic open source projects (e.g. WEKA, GATE and LingPipe ). However, what I found so far are pieces too hard to put together to fit my purpose.

What specific algorithm/software/resource do you recommend on this problem?

Thanks in advance.

I did find a tool called TagHelper which is quite suitable for few aspects of my purpose and am still exploring it.

But I still expect more insights and suggestions. Thanks!

share|improve this question
up vote 2 down vote accepted

In my limited experience short texts don't make things appreciably harder. Word count data is hopelessly sparse anyway, so vigorous regularisation of some sort is needed anyway. Basically, you've got a document classification problem with really small documents.

Since you're already looking at Lingpipe, you may find the book Text Analysis with Lingpipe helpful. It's a work in progress, but the basics of classification are there. For a more introductory exposition, without accompanying software package, there also Manning et al.'s Introduction to Information Retrieval (don't be fooled by the title).

Since 'accuracy is important' the precision, recall, and confusion matrix discussions in these texts will be important to understand. You may find that some categories simply cannot be reliably distinguished, a fact which is important to communicate and/or that one of recall and precision is much more important to you than the other, so you'll want to threshold your classifier decisions carefully.

share|improve this answer
Hi there, thanks for these very useful suggestions! Any further recommendation about the tools? E.g. you mentioned Lingpiepe, will that be more suitable for this purpose than, say, WEKA or GATE or others. Because I am not familiar with them. Thanks. – Flake Sep 4 '11 at 21:24
My impression is that GATE is more for folk interested in Information Extraction system construction. And personally I always found WEKA a bit awkward to use, and somewhat behind the times on algorithms. But these are very subjective judgements - you should try each one out. Most have tutorials you can work through. – conjugateprior Sep 5 '11 at 10:46

A very good resource that's pretty easy to understand is the O'Reilly book "Natural Language Processing with Python" by Bird, Klein and Loper. Also have a look at the NLTK toolkit that's used by that book.

share|improve this answer
Hi Dave, thanks for your reply. Though I do not have access to the book, I visited NLTK website. "Classifier" or "Categorize", etc, keywords are searched, but no relevant pages show up. Could you give more specific hints on using text classification using NLTK? – Flake Sep 4 '11 at 20:21
You can actually find the book online at Look at Chapter 6. – Dave Kincaid Sep 4 '11 at 21:21

I've used SVMLight for a similar text classification problem!

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.