0
$\begingroup$

In Where does Jensen's Inequality come into the EM derivation? and in McLachlan & Krishnan (1997) - The EM Algorithm and Extensions, it was shown that

\begin{equation} H(\theta|\theta^{(t)}) \leq H(\theta^{(t)} | \theta^{(t)} ) \end{equation} by Jensen's inequality. \begin{align} H(\phi' \mid \phi) - H(\phi \mid \phi) & = E[\log k(x \mid y, \phi') \mid y, \phi] - E[\log k(x \mid y, \phi) \mid y, \phi] \\ & = E[\log \{k(x \mid y, \phi') / k(x \mid y, \phi) \} \mid y, \phi] \\ & \leq \log \{ E_{\phi}[k(x \mid y, \phi') / k(x \mid y, \phi) \mid y, \phi] \} \\ & = \log \int_{\mathcal{X}(y)} k(x \mid y, \phi') dx \\ & = 0 \end{align}

I am having difficulty understanding how we go from the 3rd to 4th equality. I tried taking out $K(x \mid y, \phi)$, ie: \begin{align} \log \{ E_{\phi}[k(x \mid y, \phi') / k(x \mid y, \phi) \mid y, \phi] \} &= \log \{ \frac{1}{K(x \mid y, \phi)}E_{\phi}[k(x \mid y, \phi') \mid y, \phi] \} \\ \end{align} and also have difficulty how we went from taking expectation with respect to $\phi$ to integrating over $\mathcal{X}(y)$.

Apologize if this is a fundamental question.

$\endgroup$
1
  • $\begingroup$ Jensen's inequality implies that $\log\{\mathbb E[Z]\}\le\{\mathbb E[\log Z]\}$ whatever the (positive) random variable $Z$ and explains the move from second to third row in the equation. $\endgroup$
    – Xi'an
    Sep 8, 2021 at 16:58

1 Answer 1

2
$\begingroup$

The expectation $E_{\phi}\left [\frac{k(X \mid y, \phi')}{ k(X \mid y, \phi)} \Big| y, \phi \right ] $ is taken with respect to $x$ when $X \sim k(x \mid y, \phi)$

Thus,

\begin{align*} E_{\phi}\left [\frac{k(X \mid y, \phi')}{ k(X \mid y, \phi)} \Big| y, \phi \right ] &= \int_{{\mathcal{X}(y)}} \frac{k(x \mid y, \phi')}{ k(x \mid y, \phi)} k(x \mid y, \phi) dx \\ &= \int_{{\mathcal{X}(y)}} k(x \mid y, \phi') dx \\ &= 1 \end{align*}

For the notation ${\mathcal{X}(y)}$ my guess is that it is used to emphasize that we are integrating over the distribution of $X \mid Y=y$ and the support of this distribution may depend on $y$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.