Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 4 down vote favorite
2

Inspired by this question, I'm wondering whether any work has been done on topic models for large collections of extremely short texts. My intuition is that Twitter should be a natural inspiration for such models. However, from some limited experimentation, it looks like standard topic models (LDA, etc) perform quite poorly on this kind of data.

Does anyone out there know of any work which has been done in this area? This paper talks about applying LDA to Twitter, but I'm really interested in whether there are other algorithms which perform better in the short-document context.

share|improve this question
Twitter is a particularly difficult dataset for topic modeling not only due to the small size of the 'documents', but also due to the type of text. People tend to use various texting shorthands which makes identifying co-occurrences even more difficult. – Nick Mar 30 '12 at 22:57

1 Answer

up vote 3 down vote

While I'm not super familiar with his work, I know Jacob Eisenstein has done work in text analysis and graphical models in twitter data. In particular, this paper describes an application of topic modeling in twitter data and microblogs.

Edit: actually after reading the paper a bit more, they state:

However, the average message on Twitter is only sixteen word tokens, which is too sparse for traditional topic modeling; instead, we gathered together all of the messages from a given user into a single document.

So perhaps that very paper may not be of much help, still maybe other Eisenstein publications may lead you in the right direction.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.