The M step of EM algorithm in chapter 11.4.7 Machine learning - a probabilistic perspective is denoted: $$ \theta^{t+1} =\underset{\theta}{argmax}Q(\theta, \theta^{t})=\underset{\theta}{argmax}\sum_{i}E_{q_{i}^{t}}[logp(x_i, z_i|\theta)]$$ $\theta$ is the parameter,$t$ is the step, $Q$ is the expectation, $q_{z_i}$ is a arbitrary distribution over the hidden varaiable $z_i$,$p(x_i,z_i)$ is the distribution of the complete data.
The expectation can be written as: $$ Q(\theta, \theta^t)=\sum_{i}E_{q_{i}^{t}}[logp(x_i, z_i|\theta)] =\sum_{i}\sum_{z_i}q(z_i^t)logp(x_i, z_i| \theta) $$ In the E step: $$q(z_i^t)=p(z_i^t|x_i, \theta^{t})$$

$$Q(\theta, \theta^t)= \sum_{i}\sum_{z_i}p(z_i^t|x_i,\theta^t)logp(x_i, z_i| \theta) \quad \quad(1)$$

The M step in the chapter 9.4 PRML is denoted: $$ Q(\theta, \theta^{old})=\sum_{Z}q(Z)ln(p(X,Z|\theta))=\sum_{Z}P(Z|X|\theta^{old})lnp(X,Z|\theta) $$ $X$ is all of the observed variables, $Z$ is all of the hidden variables.
In the E step, $$ q(Z)=p(Z|X, \theta^{old})=\frac{p(X,Z|\theta^{old})}{\sum_{Z} p(X,Z|\theta^{old})}=\frac{\prod_{n=1}^{N}p(x_n, z_n|\theta^{old})}{\sum_Z\prod_{n=1}^{N}p(x_n, z_n|\theta^{old})}=\prod_{n=1}^{N}p(z_n|x_n, \theta^{old})$$ then: $$Q(\theta,\theta^{old})=\sum_{Z}\prod_{n=1}^{N}p(z_n|x_n, \theta^{old})\sum_{n=1}^{N}lnp(x_n, z_n|\theta) \quad \quad(2)$$

I don't understand why the equation (1) and equation(2) is different?

  • $\begingroup$ Are they different? (a*b)(c+d) == ac+ad+bc+bd. This is exactly the same case, just with N components rather than 2. $\endgroup$ – jkm Nov 15 '19 at 12:45
  • $\begingroup$ @jkm: There is an extra $\prod_{n=1}^{N}$ term in the second equation, and hence, it is not obvious how both the equations are equal. $\endgroup$ – honeybadger Nov 15 '19 at 14:04
  • $\begingroup$ I'm aware. My point is that it follows from the basic algebraic properties of the binary operators over reals, so you can prove the same thing with N=2 and generalize. Your proof amounts to mine for N=2. $\endgroup$ – jkm Nov 15 '19 at 15:01

With respect to Equation 2, \begin{align}Q(\theta,\theta^{old}) & = \sum_{Z} \prod_{n=1}^{N}p(z_n|x_n, \theta^{old})\sum_{n=1}^{N}\log p(x_n, z_n|\theta) \quad \quad(3)\\ & = \sum_{z_1} \sum_{z_2} \dots \sum_{z_N} \prod_{n=1}^{N}p(z_n|x_n, \theta^{old}) \sum_{n=1}^{N}\log p(x_n, z_n|\theta) \quad \quad(4)\\ &= \sum_{z_1} p(z_1|x_1, \theta^{old})\log p(x_1, z_1|\theta) + \sum_{z_2} p(z_2|x_2, \theta^{old})\log p(x_2, z_2|\theta) \dots \quad \quad(5) \\ & = \sum_{i=1}^{N} \sum_{z_i} p(z_i|x_i)\log p(x_i, z_i|\theta) \\ \\ \end{align}

Here we obtained $(5)$, using the fact that $$\sum_{Z} \prod_{n=1}^{N}p(z_n|x_n, \theta^{old}) \sum_{n=1}^{N} \log p(x_n,z_n| \theta ) = \\ \sum_{z_1} \dots \sum_{z_N} \prod_{n=1}^{N}p(z_n|x_n, \theta^{old}) (\log p(x_1,z_1| \theta ) + \log p(x_2,z_2| \theta ) + \dots) \\ = \\ \sum_{z_1} \dots \sum_{z_N} \prod_{n=1}^{N}p(z_n|x_n, \theta^{old}) (\log p(x_1,z_1| \theta ) ) + \sum_{z_1} \dots \sum_{z_N} \prod_{n=1}^{N}p(z_n|x_n, \theta^{old}) (\log p(x_2,z_2| \theta ) ) + \dots = \sum_{z_1} p(z_1|x_1, \theta^{old})\log p(x_1, z_1|\theta) + \sum_{z_2} p(z_2|x_2, \theta^{old})\log p(x_2, z_2|\theta)$$

| cite | improve this answer | |
  • $\begingroup$ Thanks for your detail explain. Should the equation below equation (5) be $\sum_{i=1}^{N}\sum_{z_i}p(z_i|x_i)logp(x_i,z_i|\theta)$ ? $\endgroup$ – Tengfei Tian Nov 17 '19 at 7:06
  • $\begingroup$ @TengfeiTian: Thanks for pointing out the typo. Fixed it. $\endgroup$ – honeybadger Nov 17 '19 at 11:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.