3
$\begingroup$

I am new in the probabilistic topic modeling, and I need to understand deeply the Latent Dirichlet Allocation (LDA) process.

I understand that what is the inference process in LDA, and there are 2 "types" of inference: probabilistic methods (like the Gibbs sampling) and deterministic ones.

In the deterministic type, some papers talk about Expectation Maximization algorithm and others talk about something called Variational Inference.

So my questions are

  1. Are those terms or concepts same?
  2. How EM algorithm works in LDA?
$\endgroup$

1 Answer 1

5
$\begingroup$

First, I notice that the answer given by AdamO discusses linear discriminant analysis. Since the question mentions topic modeling, I believe it is about latent Dirichlet allocation instead.

Now to answer the question:

  1. EM and variational inference are not the same. In EM, you maximize the likelihood or posterior wrt. the parameters with the hidden variables marginalized. In VI, the parameters are also regarded as hidden variables, and you want to approximate the posterior of the hidden variables by a variational distribution. You may think of EM as a special case of mean-field VI where the variational distributions are assumed to be point estimations.

  2. To apply EM to LDA: In the E-step, you fix $\theta$ (the topic distribution of the document) and $\phi$ (the word distribution under a topic) and compute the distribution $q(z)$ of $z$ (the topic assignment of each word). In the M-step, you update $\theta$ and $\phi$ to optimize the expected log likelihood, where the expectation is taken based on $q(z)$.

$\endgroup$
2
  • 1
    $\begingroup$ I would say that EM is a class of algorithms for optimisation, and variational inference is an approach to turning inferential problems into optimisation problems. You could (in principle) use an EM-type algorithm to solve an optimisation problem created by variational inference. $\endgroup$ Jul 20, 2020 at 7:08
  • $\begingroup$ to my understanding, the standard implementation of LDA follows variational inference, not EM. However, the difference might come off as very subtle as the outcome of both the processes are pretty much the same. $\endgroup$
    – Arighna
    Dec 7, 2021 at 15:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.