46

The answer depends on whether you are assuming the symmetric or asymmetric dirichlet distribution (or, more technically, whether the base measure is uniform). Unless something else is specified, most implementations of LDA assume the distribution is symmetric. For the symmetric distribution, a high alpha-value means that each document is likely to contain a ...


32

Recently, a huge body of literature discussing how to extract information from written text has grown. Hence I will just describe four milestones/popular models and their advantages/disadvantages and thus highlight (some of) the main differences (or at least what I think are the main/most important differences). You mention the "easiest" approach, which ...


26

Implementation: The topicmodels package provides an interface to the GSL C and C++ code for topic models by Blei et al. and Phan et al. For the earlier it uses Variational EM, for the latter Gibbs Sampling. See http://www.jstatsoft.org/v40/i13/paper . The package works well with the utilities from the tm package. The lda package uses a collapsed Gibbs ...


17

This is indeed something often glossed over. Some people are doing something a bit cheeky: holding out a proportion of the words in each document, and giving using predictive probabilities of these held-out words given the document-topic mixtures as well as the topic-word mixtures. This is obviously not ideal as it doesn't evaluate performance on any held-...


12

+1 for topicmodels. @Momo's answer is very comprehensive. I'd just add that topicmodels takes input as document term matrices, which are easily made with the tm package or using python. The lda package uses a more esoteric form of input (based on Blei's LDA-C) and I've had no luck using the built-in functions to convert dtm into the lda package format (the ...


12

Can LDA be used to detect the topic of A SINGLE document? Yes, in its particular representation of 'topic,' and given a training corpus of (usually related) documents. LDA represents topics as distributions over words, and documents as distributions over topics. That is, one very purpose of LDA is to arrive at probabilistic representation of each document ...


11

David Blei has a great talk introducing LDA to students of a summer class: http://videolectures.net/mlss09uk_blei_tm/ In the first video he covers extensively the basic idea of topic modelling and how Dirichlet distribution come into play. The plate notation is explained as if all hidden variables are observed to show the dependencies. Basically topics are ...


10

The notion of "topics" in so-called "topic models" is misleading. The model does not know or is not designed to know semantically coherent "topics" at all. The "topics" are just distributions over tokens (words). In other words, the model just capture the high-order co-occurrence of terms. Whether these structures mean something or not is not the purpose of ...


10

Andrew Ng is indeed one of the co-authors on the paper that first introduced LDA. David Blei is the first author of that same paper and gave two fantastic 90-minute lectures on the topic back in 2009. They are more approachable than the actual journal article itself and are worth watching if you want to do anything with LDA.


9

I think it is not really a question of better and worse but what data you have available and interpretability. If the data you have is at least partially labeled, whether with something like traditional topic classes, or something like hashtags, then labeled LDA may be interesting to pursue, otherwise not. Doing labeled LDA is certain to mean that the ...


8

Typically, in the context of Latent Dirichlet Allocation (used for Topic Modeling), we assume that the documents come from a generative process. I'll avoid math notation. Look at this figure: (1) Every topic is generated from a Dirichlet distribution of $V$ dimensions where $V$ is the size of your vocabulary. (2) For every document: (2.1) Generate a ...


7

This is a late answer, but it can be useful for other people searching for related research and tools for this problem: Weiwei Guo from Columbia implemented code for short-text topic modeling. He described the implementation in the paper "Modeling Sentences in the Latent Space" (http://aclweb.org/anthology-new/P/P12/P12-1091v2.pdf) and the code is available ...


6

While I'm not super familiar with his work, I know Jacob Eisenstein has done work in text analysis and graphical models in twitter data. In particular, this paper describes an application of topic modeling in twitter data and microblogs. Edit: actually after reading the paper a bit more, they state: However, the average message on Twitter is only ...


6

A recent paper called "a biterm topic model for short text" (WWW13) has made some progress on this topic, and here is its code


6

For my own curiosity, I applied a clustering algorithm that I've been working on to this dataset. I've temporarily put-up the results here (choose the essays dataset). It seems like the problem is not the starting points or the algorithm, but the data. You can 'reasonably' (subjectively, in my limited experience) get good clusters even with 147 instances ...


6

You might compute PMI using Wikipedia, as following: 1) Using Lucene to index a Wikipedia dump 2) Using Lucene API, it is straightforward to get: The number (N1) of documents containing word1 and the number (N2) of documents containing word2. So, Prob(word1) = (N1 + 1) / N and Prob(word2) = (N2 + 1) / N, where N is the total number of documents in ...


6

It is important to remember that topic models such as LDA were primarily developed for unsupervised text summarization. So often, there is not a "best" choice for how many top words to show. Most research papers on topic models tend to use the top 5-20 words. If you use more than 20 words, then you start to defeat the purpose of succinctly summarizing the ...


5

LDA can capture higher-order of co-occurrences of terms (due to the assumption of each topic is a multinomial distribution over terms), which is not possible by just computing PMI between terms.


5

One disadvantage of an unsupervised method like LDA is it will generally take considerably longer to train compared to supervised methods. I'm also confused about the 2% increase you mention, based on table 2 it looks like an 8% difference between the best supervised approach they compared against and their best unsupervised model. While I generally like ...


5

I might be 3 years late but I want to follow up your question on the example of "high-order of co-occurrences". Basically, if term t1 co-occurs with term t2 that co-occurs with term t3, then term t1 is the 2nd-order co-occurrence with term t3. You can go to higher order if you want but at the end you control how similar two words should be.


5

Briefly, my answer is "yes". The result of any LDA inference algorithm is $\theta_{d,t}$ and $\phi_{t,w}$, distribution of topics in each document and distribution of terms in each topics. Given these distributions, one can obtain estimate for $p(z|d,w)$, conditional probability of a topic $z$ for word $w$ in document $d$: $$ p(z|d,w)=\frac{p(z,d,w)}{\sum\...


5

You have to keep in mind that the vocabulary size is not the only factor which will affect the LSA results. In my experience a lot depends on the inflectional properties of a language. English is very LSA-friendly in this respect as it has virtually no inflection. Some Romance languages on the other hand still have pretty rich inflection so that having ...


4

Typical value of alpha which is used in practice is 50/T where T is number of topics and value of beta is 0.1 or 200/W , where W is number of words in vocabulary.


4

The mean and variance of a Gaussian are the unknown parameters that specify that distribution in that case. Likewise, in topic modeling, you attempt to learn the unknown parameters of $K$ topics, where each topic is a multinomial distribution over words in the vocabulary. Thus, it is the parameters of each multinomial distribution (each topic) that you seek ...


4

You can use the original C++ code of the LDA inventors Blei et al. Also quite fast is GibbsLDA (written in C and C++). If you want to parallelize it you might want to check out plda (C++).


4

Under the assumed generative model, each topic $z_n$ indexes a distribution over words in the topic, and each $z_n$ is randomly drawn from $Multinomial(\theta)$. Per Blei et al, "The basic idea is that documents are represented as random mixtures over latent topics, where each topic is characterized as a distribution over words." The keyword is "latent," i....


4

The approach termed "Type 1" is already explored in the paper by Blei et al. (2003) http://jmlr.org/papers/volume3/blei03a/blei03a.pdf in §7.2. The result is that this approach is valuable in feature selection for the SVM. So "Type 1" is definitely one right way. I have no comments "Type 2" except that it lacks the clarity that "Type 1" has.


4

I would not use Gaussian mixture models, as they require the constituent distributions to all be normal. You have counts, so GMM is inappropriate by definition. Latent Dirichlet allocation (full disclosure: I don't really know topic modeling) requires your data to be multinomial, but you can have counts in that case—they would be counts of ...


4

My first bet would be that the function words in a corpus of source code differ vastly from those of standard stop lists, and that your model's first topic is indeed capturing standard programming fare: if, int, new, while, etc. Besides building a custom stop list—seeing which words have high probability under the most frequently assigned topics is a good ...


4

Counter intuitively, it appears that the log_perplexity function doesn't output a $perplexity$ after all (the documentation of the function wasn't clear enough for me personally), but a likelihood $bound$ which must be utilised in the perplexity's lower bound equation thus (Taken from this paper - Online Learning for Latent Dirichlet Allocation by Hoffman, ...


Only top voted, non community-wiki answers of a minimum length are eligible