0
$\begingroup$

Why exactly, when learning hidden variables distribution in LDA (Latent Dirichlet Allocation), one cannot use the EM (Expectation Maximization) algorithm and have to resort to a variational EM typically (or sampling method)?

Variational inference was the original method proposed for parameter inference in the 2003 paper from Blei and Al.

$\endgroup$
  • $\begingroup$ Can you add a reference to the source discussing variational EM for LDA ? $\endgroup$ – wij Mar 30 '15 at 15:39
  • $\begingroup$ Yes ! Variationnal inference was the original method proposed for parameter inference in the 2003 paper from Blei and Al: machinelearning.wustl.edu/mlpapers/paper_files/BleiNJ03.pdf $\endgroup$ – dtrckd Mar 31 '15 at 15:14
1
$\begingroup$

I do not see why you cannot use EM for LDA. To apply EM to LDA: In the E-step, you fix $\theta$ (the topic distribution of the document) and $\phi$ (the word distribution under a topic) and compute the distribution $q(z)=p(z|x,\theta,\phi)$ ($z$ is the topic assignment of each word). In the M-step, you update $\theta$ and $\phi$ to optimize the expected log likelihood, where the expectation is taken based on $q(z)$.

Of course, if you use less-than-one hyperparameters for the Dirichlet distributions, then you cannot use EM because in the M-step the expected log likelihood would contain Dirichlet over $\theta$ and $\phi$ that may have multiple modes (i.e., some of the parameters of the Dirichlet may be less than 1, leading to infinite probability density at the corners/edges of the simplex, as shown below).

$\endgroup$
  • $\begingroup$ Why the expected log likelihood "may" have multiple mode ? Under what conditions it won't ? Why does infinite means in term of number of modes ? Could you add some reference ? Thank you. $\endgroup$ – dtrckd Mar 21 '17 at 13:45
  • $\begingroup$ @dtrckd I've updated my answer to clarify this. $\endgroup$ – took Mar 22 '17 at 10:46
  • $\begingroup$ For the first part of your answer, it seems, that this is not EM, but what we could call Variational EM since we don't directly work with p(x) but q(x) instead. For the second part, I suppose you chose a Dirichlet with all hyperparameters near zeros, in this case it is like having an uniform prior. But regardless of the "less-than-one" hyperperameters settings, I can run an inference based in the "variational EM", but still not an EM one ? $\endgroup$ – dtrckd Mar 22 '17 at 16:41
  • $\begingroup$ @dtrckd The first part is indeed EM, not VEM. Here $q(z)=p(z|x,\theta,\phi)$. For the second part, Dirichlet is uniform when all the parameters are ones, not near zeros; in the near-zero case, you cannot run EM. $\endgroup$ – took Mar 23 '17 at 9:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.