I am new in the probabilistic topic modeling, and I need to understand deeply the Latent Dirichlet Allocation (LDA) process.

I understand what want to do the inference process in LDA, and I understand too that there are 2 "types" of inference: probabilistic methods (like the Gibbs sampling) and deterministic ones.

In the deterministic type some papers talk about Expectation Maximization algorithm and some others talk about something called "variational inference". So, are those things the same?

Also, can you explain how is applied the EM algorithm for the LDA?


First, I notice that the answer given by AdamO discusses linear discriminant analysis. Since the question mentions topic modeling, I believe it is about latent Dirichlet allocation instead.

Now to answer the question:

  1. EM and variational inference are not the same. In EM, you maximize the likelihood or posterior wrt. the parameters with the hidden variables marginalized. In VI, the parameters are also regarded as hidden variables, and you want to approximate the posterior of the hidden variables by a variational distribution. You may think of EM as a special case of mean-field VI where the variational distributions are assumed to be point estimations.

  2. To apply EM to LDA: In the E-step, you fix $\theta$ (the topic distribution of the document) and $\phi$ (the word distribution under a topic) and compute the distribution $q(z)$ of $z$ (the topic assignment of each word). In the M-step, you update $\theta$ and $\phi$ to optimize the expected log likelihood, where the expectation is taken based on $q(z)$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.