I am trying to understand the EM algorithm, and I see that in the expectation step,we need to compute $E_{Z|X} {\log p(X,Z)} = \sum_{Z} {p(Z|X) p(X,Z)}$ where $Z$ is the hidden variable and $X$ is the observed variable. I am understanding that we need to compute the expected value of $\log p(X,Z)$ since we cannot compute the exact value of $\log p(X,Z)$ due to the variable $Z$ whose value we are not aware of. But I have 2 separate confusions here:

1) Why don't we do, say, a block coordinate descent instead? That is, why don't we compute $Z$ and the parameters of the model (say $\theta$) iteratively such that we compute $Z$ in one step using the current value of $\theta$ and $\theta$ in the other step using the current value of $Z$?

2) Let's say I understood that we cannot do block coordinate descent but we need to compute that expectation of $\log p(X,Z)$. Then why don't we compute $E_Z {\log p(X,Z)} = \sum_{Z} {p(Z) p(X,Z)}$ but we compute $E_{Z|X} {\log p(X,Z)}$? What does it mean (intuitively, or in words) to compute an expected value with respect to a conditional probability?

  • $\begingroup$ I think in one of my unanswered question you may find your answer. $\endgroup$ – Alice May 3 '17 at 3:43
  • $\begingroup$ Here is my question stats.stackexchange.com/questions/277073/… please read through the posted photo. $\endgroup$ – Alice May 3 '17 at 3:45
  • $\begingroup$ I think the first two paragraphs are belong to your second question. I really confused about it, but may it will be clear for you. $\endgroup$ – Alice May 3 '17 at 3:46

For your first question: in EM you are alternating between updating $Z$ and updating $\theta$. (Note however that $Z$ is commonly a set of categorical variables, i.e. labels, in EM applications, so it is not exactly a "gradient descent" per se.)

For the second question, the most common example of a conditional expectation is probably the standard regression problem, i.e. $$y=f(x)+\epsilon\,,\,\mathbb{E}\big[\epsilon\big]=0\implies\mathbb{E}\big[y\mid{x}\big]=f(x)$$

With regards to the first question, I would recommend checking out the Wikipedia description. You could "update $Z$" based on the current $\theta$ (called hard EM), but in general you can update the entire posterior $p\big[Z\mid{X,\theta}\big]$ (note that the observed data $X$ is fixed throughout, see also here). For a concrete example of the difference, see here.

With regards to the requested clarification on the second question: Imagine you want to do standard MLE for $\theta$ based on the data $\hat{X}$. But the data $\hat{X}$ is not fully observed, that is, $\hat{X}=[X,Z]$ but $Z$ is unknown. As noted in Wikipedia, a natural approach would be to maximize the marginal likelihood $$p\big[X\mid{\theta}\big]=\sum_Zp\big[\hat{X}\mid{\theta}\big]$$ over possible $Z$ values. Since $X$ is known and fixed, EM proceeds by taking the expected value of the likelihood over possible $Z$ values, conditioned on $X$ and $\theta$.

Note that $p\big[Z\big]=\int{p}\big[Z\mid{X}\big]p\big[X\big]dX$. But since $X$ is known and fixed, we don't care what the likelihood of $Z$ is given some other hypothetical value of $X$. (For simplicity I omit the $p\big[\ldots\mid{\theta}\big]$ in the above.)

| cite | improve this answer | |
  • $\begingroup$ Thanks, but I am confused. Followup questions: 1) We are not updating the value of $Z$ at all, right? We are just updating the expected value of log likelihood $E(\log p(Z,X))$. Do we need to recompute $P(Z|X)$ in the next expectation step to recompute $E(\log P(Z,X))$? Even if we do that, we are not updating the actual value of $Z$ in the expectation step. 2) I can't see the relation of this example to $E_{Z|X}{\log p(Z,X)}$. The condition is not a subscript in your example, right? And the main point of my question is why don't we compute $E_Z{\log p(Z,X)}$ instead of $E_{Z|X}{\log p(Z,X)}$. $\endgroup$ – user5054 May 3 '17 at 2:14
  • $\begingroup$ You want to get the value of Z using your data. $\endgroup$ – Alice May 3 '17 at 3:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.