I am working my way through the original EM paper Maximum Likelihood from Incomplete Data by Dempster, et al.

I have run into a problem with a statement made in section 3. "General Properties". Specifically I am having a difficult time with Lemma 1:

enter image description here

My understanding of Jensen's inequality is

$$ \mathop{E}[f(X)] \geq f(\mathop{E}[X]) \text{ for convex functions } f(x) $$

or the reverse for concave functions (e.g. the log of a pdf).

If I am interpreting equation $3.3$ correctly, we have

$$ \mathop{E}[\log k(x|y, \phi')|y, \phi] \leq \mathop{E}[\log k(x|y, \phi)|y, \phi] \\ \implies \int_X \log k(x|y, \phi') p(x | y, \phi) dx \leq \int_X \log k(x|y, \phi) p(x | y, \phi) dx $$

I am failing to see where Jensen's inequality is applied as both sides of the equation include the function $\log k(\cdot)$ inside of the expectation.

Am I misunderstanding the application of Jensen's inequality?


1 Answer 1


This is the proof provided in McLachlan & Krishnan (1997) - The EM Algorithm and Extensions (converted to use the same notation as DLR):

$$ \begin{align} H(\phi' \mid \phi) - H(\phi \mid \phi) & = E[\log k(x \mid y, \phi') \mid y, \phi] - E[\log k(x \mid y, \phi) \mid y, \phi] \\ & = E[\log \{k(x \mid y, \phi') / k(x \mid y, \phi) \} \mid y, \phi] \\ & \leq \log \{ E[k(x \mid y, \phi') / k(x \mid y, \phi) \mid y, \phi] \} \\ & = \log \int_{\mathcal{X}(y)} k(x \mid y, \phi') dx \\ & = 0 \end{align} $$

Jensen's inequality is used in moving from the second to the third lines.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.