My question is:

  1. Why the incomplete-data likelihood equal to formula 1?

  2. Why it should not equal to formula 2?

I apologize for not being word-perfect in English.

I'm reading the book Pattern Recognition And Machine Learning. And I was confused at the derivation of EM algorithm in General at Page 467.

If we denote all of the observed variables by X and all of the hidden variables by Z. Our goal is to maximize the likelihood function: $p(X\mid\theta)$.

Why our goal is equal to $$\sum_Z p(X,Z \mid \theta) \qquad (1)$$


$$\sum_z p(X,Z \mid \theta) = \sum_z \prod_n \sum_k z_n^k p(x_n\mid \mu_k, \Sigma_k)\pi_k$$

from cos 513: mixture models and em algorithm page 4. As Gaussian mixture model, $p(x_n \mid \theta)= \sum_k \pi_k p(x_n \mid \mu_k, \Sigma_k)$ means instance $x_n$ with probability $\pi_k$ generated by kth Gaussian component.

And $z_n^k$ is a one of K variable, if instance $x_n$ was generated by kth Gaussian component, $z_n^k = 1$; otherwise, $z_n^k=0$.

In my mind,

$$p(X\mid\theta) = \prod_n \sum_z \sum_k z_n^k p(x_n\mid \mu_k, \Sigma_k)\pi_k \qquad (2)$$

So, I think this was the marginal likelihood. Cause, likelihood should be the product of probabilities of observations.

  • $\begingroup$ OK, think I got the answer. $Z$ here is a vector, dimension equal the number of samples N as the COS513 described. So if $z_n$ has 2 possible values, $Z$ will be $2^N$ combinations. $\endgroup$
    – roachsinai
    Commented Jul 30, 2016 at 10:11
  • $\begingroup$ I think you're right :) $\endgroup$
    – dontloo
    Commented Jul 30, 2016 at 10:26
  • $\begingroup$ Thanks. But I think formula $2$ is correct, too. For $z$ has different meaning. $\endgroup$
    – roachsinai
    Commented Jul 30, 2016 at 12:30

1 Answer 1


I think (1) and (2) are equal, with different interpretations of the $z$ that we are summing over.

For clarity I'll rewrite the formulas as $$p(X \mid \theta) = \sum_\mathbf{Z} p(X,Z \mid \theta) = \sum_\mathbf{Z} \prod_N a_n^k \qquad (1)$$ $$p(X \mid \theta) = \prod_N \sum_\mathbf{z} a_n^k \qquad (2)$$ $$a_n^k=\sum_K z_n^k p(x_n\mid\mu_k, \Sigma_k)\pi_k$$

As you well noted, in (1) there are $K^N$ different values in $\mathbf{Z}$, well in (2) there are $K$ different values in $\mathbf{z}$. Note that because $z$ is a one-hot vector, the number of possible values for $z$ is equal to the dimension of $z$ (which is the other $K$ that we are summing over in $a_n^k$).

Expanding (2) $$\prod_N \sum_\mathbf{z} a_n^k=\prod_N(a_n^1+a_n^2+...+a_n^K)=(a_1^1a_2^1...a_N^1)+(a_1^2a_2^1...a_N^1)+...+(a_1^Ka_2^K...a_N^K)=\sum_\mathbf{Z} \prod_N a_n^k$$

In fact as $z$ being one-hot, we have $$p(X \mid \theta) = \prod_N \sum_\mathbf{z} a_n^k=\prod_N \sum_\mathbf{z} \sum_K z_n^k p(x_n\mid\mu_k, \Sigma_k)\pi_k=\prod_N\sum_K\pi_kp(x_n\mid\mu_k, \Sigma_k)$$ which is the original definition. So the above just shows that integrating over the joint probability equals the marginal probability, $p(X \mid \theta) = \sum_\mathbf{Z} p(X,Z \mid \theta)$.

It feels like there should be one more subscript or superscript of the $z_n^k$ term in the original formula to denote which one-hot vector $z$ it is.

  • $\begingroup$ Can't agree more! Thanks for your clarification. I have voted your answer, but stats.stackexchange.com will not display it for my reputations less than 15 :-(. $\endgroup$
    – roachsinai
    Commented Jul 31, 2016 at 0:29
  • $\begingroup$ @roachsinai you're welcome, yes the site requires a minimum reputation for that, also there's an "accept" button just below there, if you find this answer acceptable :P $\endgroup$
    – dontloo
    Commented Jul 31, 2016 at 5:36
  • 1
    $\begingroup$ Yes! Got it! :-) $\endgroup$
    – roachsinai
    Commented Jul 31, 2016 at 9:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.