Is there a special case of the EM algorithm for exponential family distributions?

Question

According to Wikipedia, the formal definition of the EM algorithm is

The EM algorithm seeks to find the MLE of the marginal likelihood by iteratively applying these two steps:

Expectation step (E step): Define $Q(\boldsymbol\theta\mid\boldsymbol\theta^{(t)})$ as the expected value of the log likelihood function of $\boldsymbol\theta$, with respect to the current conditional distribution of $\mathbf{Z}$ given $\mathbf{X}$ and the current estimates of the parameters $\boldsymbol\theta^{(t)}$: $$ Q(\boldsymbol\theta\mid\boldsymbol\theta^{(t)}) = \operatorname{E}_{\mathbf{Z}\mid\mathbf{X},\boldsymbol\theta^{(t)}}\left[ \log L (\boldsymbol\theta; \mathbf{X},\mathbf{Z}) \right] \, $$

Maximization step (M step): Find the parameters that maximize this quantity: $$ \boldsymbol\theta^{(t+1)} = \underset{\boldsymbol\theta}{\operatorname{arg\,max}} \ Q(\boldsymbol\theta\mid\boldsymbol\theta^{(t)}) \, $$

In instances where $p(\mathbf{x},\mathbf{z})$ is an exponential family distribution, I have found that authors used $$ Q(\boldsymbol\theta\mid\boldsymbol\theta^{(t)}) = \log L (\boldsymbol\theta; \mathbf{X},\operatorname{E}_{\mathbf{Z}\mid\mathbf{X},\boldsymbol\theta^{(t)}}\left[\mathbf{Z}\right]) \, $$ instead of $$ Q(\boldsymbol\theta\mid\boldsymbol\theta^{(t)}) = \operatorname{E}_{\mathbf{Z}\mid\mathbf{X},\boldsymbol\theta^{(t)}}\left[ \log L (\boldsymbol\theta; \mathbf{X},\mathbf{Z}) \right] \, $$ where $\operatorname{E}_{\mathbf{Z}\mid\mathbf{X},\boldsymbol\theta^{(t)}}\left[\mathbf{Z}\right]$ is the estimated data. Are these two equations equivalent?

Answer 1

Suppose we want to estimate the parameter $\theta$ of the distribution $p(z;\theta)$ of a random variable $Z$. If we had samples $z_1, z_2, \dots, z_N$, we could easily do this using maximum likelihood estimation. However, suppose instead we are given the samples $x_1,x_2,\dots,x_N$ of another random variable $X$, where $X$ and $Z$ are related via the joint distribution $p(x,z)$. Since $p(x,z) = p(x \mid z) \cdot p(z;\theta)$, then $p(x,z) = p(x,z;\theta)$.

We could then define the complete-data likelihood as $$ p(x_1,x_2,\dots,x_N,z_1,z_2,\dots,z_N) $$ We can also assume that each pair $(x_i,z_i)$ is independent of and identically distributed to every other pair for $i \in \{1,2,\dots,N\}$ such that $$ p(x_1,x_2,\dots,x_N,z_1,z_2,\dots,z_N) = \prod_{i=1}^N p(x_i,z_i;\theta) $$ and $$ \log p(x_1,x_2,\dots,x_N,z_1,z_2,\dots,z_N) = \sum_{i=1}^N \log p(x_i,z_i;\theta) $$ Next, let $$ p(x_i,z_i;\theta) = h(x_i,z_i) \cdot \exp(\eta (\theta) \cdot T(x_i,z_i) + A(\theta)) $$ such that $$ \log p(x_i,z_i;\theta) = \log h(x_i,z_i) + \eta (\theta) \cdot T(x_i,z_i) + A(\theta) $$ Then, note that \begin{align} \mathbb{E}\left[\log p(x_i,z_i;\theta) \mid x_i\right] &= \mathbb{E}\left[\log h(x_i,z_i) \mid x_i\right] + \eta (\theta) \cdot \mathbb{E}\left[T(x_i,z_i) \mid x_i\right] + A(\theta) \\ \log p(x_i,\mathbb{E}\left[z_i \mid x_i\right];\theta) &= \log h(x_i,\mathbb{E}\left[z_i \mid x_i\right]) + \eta (\theta) \cdot T(x_i,\mathbb{E}\left[z_i \mid x_i\right]) + A(\theta) \end{align} and so, in general, $$ \mathbb{E}\left[\log p(x_i,z_i;\theta) \mid x_i\right] \neq \log p(x_i,\mathbb{E}\left[z_i \mid x_i\right];\theta) $$