Sparse Additive Generative Models (SAGE) v/s LDA (Latent Dirichlet Allocation) Intuitively, how are Sparse Additive Generate Models (SAGE) by Eisenstein, different from Multinomial Dirichlet distributions. I understood that in this model distributions are added in logarithmic space. However, It is not clear why it can prevail over existing topic models. Does log space has any special characteristic which makes it more suitable, especially for sparse data.
 A: First, it's important to note that topics in LDA and SAGE are not quite the same thing. In LDA, a topic is a distribution over words. In SAGE, it's a distribution over deviations from some background distribution over words.
The sparsity that SAGE refers to is in these topic distributions -- SAGE has sparsity in topic distributions over words, but LDA does not.
Say we have K topics and V words in the vocabulary. One argument for taking the SAGE approach is that we may not have confidence that all of these K times V probabilities are properly estimated (perhaps due to lack of data). While we typically only pay attention to the top 10 or 20 words per topic to get a qualitative sense of what it's about, ALL of the words and associated probabilities in a topic are used for inference. So, the sparsity in SAGE (that, again, doesn't exist in LDA) addresses this issue. One benefit of this is that if a significant portion of the probabilities in LDA are not well calibrated, then SAGE should do a better job predicting the topics of unseen documents.
SAGE accomplishes this by specifying a background distribution over V, then modeling the deviation in log-frequencies between the background distribution and a topic distribution, and insuring that many of these deviations are zero by using sparsity inducing prior.
This pdf may also be useful:
http://www.cc.gatech.edu/~jeisenst/papers/icml2011presentation.pdf
