1
$\begingroup$

In the explanation of the EM (Expectation maximization) algorithm p.328 in the book "Statistical inference" by G. Casella and R. Berger, 2nd edition, they present the following:

$\mathbf{Y} = (Y_1, ..., Y_n)$ is the observed (incomplete) data.
$\mathbf{X} = (X_1, ..., X_n)$ is the missing (augmented) data.

The EM algorithm allows us to maximize $L(\theta|\mathbf{y})$ by working with only $L(\theta|\mathbf{y}, \mathbf{x})$ and the conditional pdf or pmf of $\mathbf{X}$ given $\mathbf{y}$ and $\theta$, defined by $$ \begin{align} &L(\theta|\mathbf{y}, \mathbf{x}) = f(\mathbf{x}, \mathbf{y}|\theta)\\ &L(\theta|\mathbf{y}) = g(\mathbf{y}|\theta)\\ &k(\mathbf{x}|\theta, \mathbf{y}) = \dfrac{f(\mathbf{x}, \mathbf{y}|\theta)}{g(\mathbf{y}|\theta)} \quad (7.2.17) \end{align} $$
Rearrangement of the last equation in (7.2.17) gives the identity
$$\begin{align} \log L(\theta|\mathbf{y}) = \log L(\theta| \mathbf{y}, \mathbf{x}) - \log k(\mathbf{x}| \theta, \mathbf{y}) \quad (7.2.18) \end{align}$$ As $\mathbf{x}$ is missing data and hence not observed, we replace the right side of (7.2.18) with its expectation under $k(\mathbf{x}|\theta', \mathbf{y})$, creating the new identity
\begin{align} \log L(\theta|\mathbf{y}) = E[\log L(\theta|\mathbf{y}, \mathbf{X})|\theta', \mathbf{y}] - E[\log k(\mathbf{X}| \theta, \mathbf{y})|\theta', \mathbf{y}] \quad (7.2.19) \end{align}

I don't understand how why the RHS in (7.2.18) is equal to the RHS in (7.2.19), how can one show this?

Edit
I realize now that I am particularly confused regarding what should be seen as variables and constants in equations (7.2.17-7.2.19). The beginning of the quoted section that I added in my edit that says "pdf or pmf of $\mathbf{X}$ given $\mathbf{y}$ and $\theta$" makes it seem like $\mathbf{y}$ and $\theta$ should be treated as constants. Given this, identity (7.2.18) could be written as $$ C = h(\mathbf{x}) \quad (*) $$ where $C$ is the constant LHS of (7.2.18) and $h(\mathbf{x})$ is the RHS of (7.2.18). But since $h(\mathbf{x})$ is a constant $C$, then indeed $$ E[h(\mathbf{x})|\theta', \mathbf{y}] = E[C|\theta', \mathbf{y}] = C $$

$\endgroup$

1 Answer 1

2
$\begingroup$

You take on both sides of equation (7.2.18) the expectation w.r.t. $k(\mathbf{x}|\theta^\prime, \mathbf{y})$. Since the LHS doesn't depend on $\mathbf{x}$, nothing changes there. And the RHS of (7.2.19) is obtained because the expectation operator is linear (i.e. $E[X+Y] = E[X] + E[Y]$). So, e.g. the first term in (7.2.19): $$E[\log L(\theta|\mathbf{y}, \mathbf{X})|\theta^\prime, \mathbf{y}]$$ just means "expectation of $\log L(\theta | \mathbf{y}, \mathbf{x})$ under $k(\mathbf{x}|\theta^\prime, \mathbf{y})$".

$\endgroup$
4
  • $\begingroup$ Thank you for you answer. In one way I think you answer makes a lot of sense. My only doubt is that the RHS of (7.2.18) seems to be a function of $\mathbf{x}, \mathbf{y}$ and $\theta$ while the RHS of (7.2.19) seems to be a function of only $\mathbf{y}$ and $\theta$. But at the same time the LHS of (7.2.18) is also a function of only $\mathbf{y}$ and $\theta$. I think I need to edit my question a bit to clarify. $\endgroup$ Commented Mar 17, 2022 at 10:18
  • $\begingroup$ Yes, by taking the expectation w.r.t. some variable (in your case $\mathbf{x}$), the result is not any more dependent on that variable. $\endgroup$
    – frank
    Commented Mar 17, 2022 at 10:22
  • $\begingroup$ Right, so the RHS of (7.2.18) is just constant w.r.t $\mathbf{x}$ even though when looking at the expression alone it seems that it could be a non-constant function of $\mathbf{x}$. I've edited my question with how I understand it now. Would be greateful if you could confirm whether I've understood things correctly, but I'll accept your answer anyways. $\endgroup$ Commented Mar 17, 2022 at 10:32
  • 1
    $\begingroup$ In general: if $f(X, Y)$ is a function that depends on two random variables $X, Y$, then, taking the expectation w.r.t. one of them, e.g. $X$, then $E_X[f(X, Y)]$ gives a function that does not depend on $X$ anymore. And yes, in (2.7.18), the LHS is already constant in $\mathbf{x}$, since the two terms cancel each other's dependence on $\mathbf{x}$. $\endgroup$
    – frank
    Commented Mar 17, 2022 at 11:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.