# How to understand the difference of EM algorithm between PRML and 'Machine learning - a probabilistic perspective'

The M step of EM algorithm in chapter 11.4.7 Machine learning - a probabilistic perspective is denoted: $$\theta^{t+1} =\underset{\theta}{argmax}Q(\theta, \theta^{t})=\underset{\theta}{argmax}\sum_{i}E_{q_{i}^{t}}[logp(x_i, z_i|\theta)]$$ $$\theta$$ is the parameter,$$t$$ is the step, $$Q$$ is the expectation, $$q_{z_i}$$ is a arbitrary distribution over the hidden varaiable $$z_i$$,$$p(x_i,z_i)$$ is the distribution of the complete data.
The expectation can be written as: $$Q(\theta, \theta^t)=\sum_{i}E_{q_{i}^{t}}[logp(x_i, z_i|\theta)] =\sum_{i}\sum_{z_i}q(z_i^t)logp(x_i, z_i| \theta)$$ In the E step: $$q(z_i^t)=p(z_i^t|x_i, \theta^{t})$$

then:
$$Q(\theta, \theta^t)= \sum_{i}\sum_{z_i}p(z_i^t|x_i,\theta^t)logp(x_i, z_i| \theta) \quad \quad(1)$$

The M step in the chapter 9.4 PRML is denoted: $$Q(\theta, \theta^{old})=\sum_{Z}q(Z)ln(p(X,Z|\theta))=\sum_{Z}P(Z|X|\theta^{old})lnp(X,Z|\theta)$$ $$X$$ is all of the observed variables, $$Z$$ is all of the hidden variables.
In the E step, $$q(Z)=p(Z|X, \theta^{old})=\frac{p(X,Z|\theta^{old})}{\sum_{Z} p(X,Z|\theta^{old})}=\frac{\prod_{n=1}^{N}p(x_n, z_n|\theta^{old})}{\sum_Z\prod_{n=1}^{N}p(x_n, z_n|\theta^{old})}=\prod_{n=1}^{N}p(z_n|x_n, \theta^{old})$$ then: $$Q(\theta,\theta^{old})=\sum_{Z}\prod_{n=1}^{N}p(z_n|x_n, \theta^{old})\sum_{n=1}^{N}lnp(x_n, z_n|\theta) \quad \quad(2)$$

I don't understand why the equation (1) and equation(2) is different?

• Are they different? (a*b)(c+d) == ac+ad+bc+bd. This is exactly the same case, just with N components rather than 2. – jkm Nov 15 '19 at 12:45
• @jkm: There is an extra $\prod_{n=1}^{N}$ term in the second equation, and hence, it is not obvious how both the equations are equal. – honeybadger Nov 15 '19 at 14:04
• I'm aware. My point is that it follows from the basic algebraic properties of the binary operators over reals, so you can prove the same thing with N=2 and generalize. Your proof amounts to mine for N=2. – jkm Nov 15 '19 at 15:01

With respect to Equation 2, \begin{align}Q(\theta,\theta^{old}) & = \sum_{Z} \prod_{n=1}^{N}p(z_n|x_n, \theta^{old})\sum_{n=1}^{N}\log p(x_n, z_n|\theta) \quad \quad(3)\\ & = \sum_{z_1} \sum_{z_2} \dots \sum_{z_N} \prod_{n=1}^{N}p(z_n|x_n, \theta^{old}) \sum_{n=1}^{N}\log p(x_n, z_n|\theta) \quad \quad(4)\\ &= \sum_{z_1} p(z_1|x_1, \theta^{old})\log p(x_1, z_1|\theta) + \sum_{z_2} p(z_2|x_2, \theta^{old})\log p(x_2, z_2|\theta) \dots \quad \quad(5) \\ & = \sum_{i=1}^{N} \sum_{z_i} p(z_i|x_i)\log p(x_i, z_i|\theta) \\ \\ \end{align}
Here we obtained $$(5)$$, using the fact that $$\sum_{Z} \prod_{n=1}^{N}p(z_n|x_n, \theta^{old}) \sum_{n=1}^{N} \log p(x_n,z_n| \theta ) = \\ \sum_{z_1} \dots \sum_{z_N} \prod_{n=1}^{N}p(z_n|x_n, \theta^{old}) (\log p(x_1,z_1| \theta ) + \log p(x_2,z_2| \theta ) + \dots) \\ = \\ \sum_{z_1} \dots \sum_{z_N} \prod_{n=1}^{N}p(z_n|x_n, \theta^{old}) (\log p(x_1,z_1| \theta ) ) + \sum_{z_1} \dots \sum_{z_N} \prod_{n=1}^{N}p(z_n|x_n, \theta^{old}) (\log p(x_2,z_2| \theta ) ) + \dots = \sum_{z_1} p(z_1|x_1, \theta^{old})\log p(x_1, z_1|\theta) + \sum_{z_2} p(z_2|x_2, \theta^{old})\log p(x_2, z_2|\theta)$$
• Thanks for your detail explain. Should the equation below equation (5) be $\sum_{i=1}^{N}\sum_{z_i}p(z_i|x_i)logp(x_i,z_i|\theta)$ ? – Tengfei Tian Nov 17 '19 at 7:06