# Jensen's inequality in EM derivation: expectation derivation

In Where does Jensen's Inequality come into the EM derivation? and in McLachlan & Krishnan (1997) - The EM Algorithm and Extensions, it was shown that

$$$$H(\theta|\theta^{(t)}) \leq H(\theta^{(t)} | \theta^{(t)} )$$$$ by Jensen's inequality. \begin{align} H(\phi' \mid \phi) - H(\phi \mid \phi) & = E[\log k(x \mid y, \phi') \mid y, \phi] - E[\log k(x \mid y, \phi) \mid y, \phi] \\ & = E[\log \{k(x \mid y, \phi') / k(x \mid y, \phi) \} \mid y, \phi] \\ & \leq \log \{ E_{\phi}[k(x \mid y, \phi') / k(x \mid y, \phi) \mid y, \phi] \} \\ & = \log \int_{\mathcal{X}(y)} k(x \mid y, \phi') dx \\ & = 0 \end{align}

I am having difficulty understanding how we go from the 3rd to 4th equality. I tried taking out $$K(x \mid y, \phi)$$, ie: \begin{align} \log \{ E_{\phi}[k(x \mid y, \phi') / k(x \mid y, \phi) \mid y, \phi] \} &= \log \{ \frac{1}{K(x \mid y, \phi)}E_{\phi}[k(x \mid y, \phi') \mid y, \phi] \} \\ \end{align} and also have difficulty how we went from taking expectation with respect to $$\phi$$ to integrating over $$\mathcal{X}(y)$$.

Apologize if this is a fundamental question.

• Jensen's inequality implies that $\log\{\mathbb E[Z]\}\le\{\mathbb E[\log Z]\}$ whatever the (positive) random variable $Z$ and explains the move from second to third row in the equation. Sep 8, 2021 at 16:58

The expectation $$E_{\phi}\left [\frac{k(X \mid y, \phi')}{ k(X \mid y, \phi)} \Big| y, \phi \right ]$$ is taken with respect to $$x$$ when $$X \sim k(x \mid y, \phi)$$
\begin{align*} E_{\phi}\left [\frac{k(X \mid y, \phi')}{ k(X \mid y, \phi)} \Big| y, \phi \right ] &= \int_{{\mathcal{X}(y)}} \frac{k(x \mid y, \phi')}{ k(x \mid y, \phi)} k(x \mid y, \phi) dx \\ &= \int_{{\mathcal{X}(y)}} k(x \mid y, \phi') dx \\ &= 1 \end{align*}
For the notation $${\mathcal{X}(y)}$$ my guess is that it is used to emphasize that we are integrating over the distribution of $$X \mid Y=y$$ and the support of this distribution may depend on $$y$$