EM to Variational EM in LDA

Why exactly, when learning hidden variables distribution in LDA (Latent Dirichlet Allocation), one cannot use the EM (Expectation Maximization) algorithm and have to resort to a variational EM typically (or sampling method)?

Variational inference was the original method proposed for parameter inference in the 2003 paper from Blei and Al.

• Can you add a reference to the source discussing variational EM for LDA ? – wij Mar 30 '15 at 15:39
• Yes ! Variationnal inference was the original method proposed for parameter inference in the 2003 paper from Blei and Al: machinelearning.wustl.edu/mlpapers/paper_files/BleiNJ03.pdf – dtrckd Mar 31 '15 at 15:14

I do not see why you cannot use EM for LDA. To apply EM to LDA: In the E-step, you fix $\theta$ (the topic distribution of the document) and $\phi$ (the word distribution under a topic) and compute the distribution $q(z)=p(z|x,\theta,\phi)$ ($z$ is the topic assignment of each word). In the M-step, you update $\theta$ and $\phi$ to optimize the expected log likelihood, where the expectation is taken based on $q(z)$.
Of course, if you use less-than-one hyperparameters for the Dirichlet distributions, then you cannot use EM because in the M-step the expected log likelihood would contain Dirichlet over $\theta$ and $\phi$ that may have multiple modes (i.e., some of the parameters of the Dirichlet may be less than 1, leading to infinite probability density at the corners/edges of the simplex, as shown below). • @dtrckd The first part is indeed EM, not VEM. Here $q(z)=p(z|x,\theta,\phi)$. For the second part, Dirichlet is uniform when all the parameters are ones, not near zeros; in the near-zero case, you cannot run EM. – took Mar 23 '17 at 9:51