0
$\begingroup$

Why exactly, when learning hidden variables distribution in LDA (Latent Dirichlet Allocation), one cannot use the EM (Expectation Maximization) algorithm and have to resort to a variational EM typically (or sampling method)?

Variational inference was the original method proposed for parameter inference in the 2003 paper from Blei and Al.

$\endgroup$
2
  • $\begingroup$ Can you add a reference to the source discussing variational EM for LDA ? $\endgroup$
    – wij
    Commented Mar 30, 2015 at 15:39
  • $\begingroup$ Yes ! Variationnal inference was the original method proposed for parameter inference in the 2003 paper from Blei and Al: machinelearning.wustl.edu/mlpapers/paper_files/BleiNJ03.pdf $\endgroup$
    – dtrckd
    Commented Mar 31, 2015 at 15:14

1 Answer 1

1
$\begingroup$

I do not see why you cannot use EM for LDA. To apply EM to LDA: In the E-step, you fix $\theta$ (the topic distribution of the document) and $\phi$ (the word distribution under a topic) and compute the distribution $q(z)=p(z|x,\theta,\phi)$ ($z$ is the topic assignment of each word). In the M-step, you update $\theta$ and $\phi$ to optimize the expected log likelihood, where the expectation is taken based on $q(z)$.

Of course, if you use less-than-one hyperparameters for the Dirichlet distributions, then you cannot use EM because in the M-step the expected log likelihood would contain Dirichlet over $\theta$ and $\phi$ that may have multiple modes (i.e., some of the parameters of the Dirichlet may be less than 1, leading to infinite probability density at the corners/edges of the simplex, as shown below).

$\endgroup$
4
  • $\begingroup$ Why the expected log likelihood "may" have multiple mode ? Under what conditions it won't ? Why does infinite means in term of number of modes ? Could you add some reference ? Thank you. $\endgroup$
    – dtrckd
    Commented Mar 21, 2017 at 13:45
  • $\begingroup$ @dtrckd I've updated my answer to clarify this. $\endgroup$
    – took
    Commented Mar 22, 2017 at 10:46
  • $\begingroup$ For the first part of your answer, it seems, that this is not EM, but what we could call Variational EM since we don't directly work with p(x) but q(x) instead. For the second part, I suppose you chose a Dirichlet with all hyperparameters near zeros, in this case it is like having an uniform prior. But regardless of the "less-than-one" hyperperameters settings, I can run an inference based in the "variational EM", but still not an EM one ? $\endgroup$
    – dtrckd
    Commented Mar 22, 2017 at 16:41
  • $\begingroup$ @dtrckd The first part is indeed EM, not VEM. Here $q(z)=p(z|x,\theta,\phi)$. For the second part, Dirichlet is uniform when all the parameters are ones, not near zeros; in the near-zero case, you cannot run EM. $\endgroup$
    – took
    Commented Mar 23, 2017 at 9:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.