Why maximizing the expected value of log likelihood under the posterior distribution of latent variables maximize the observed data log-likelihood?

Question

I am trying to understand the Expectation-Maximization algorithm and I am not able to get the intuition of a particular step. I am able to verify the mathematical derivation but I want to understand the why we encounter this particular term.
In the EM algorithm, we know that our log likelihood $\ln p(X|\theta)$ can be written as $\mathcal{L}(q,\theta) + KL(q||p)$.

And $\mathcal{L}(q,\theta) = \mathcal{Q}(θ, θ^{old}) + const$ where the $const$ is the entropy of the the distribution $q(Z)= p(Z|X,θ^{old})$. And the term $\mathcal{Q}(θ, θ^{old})$ represents the expectation of the complete-data log likelihood under the posterior distribution $p(Z|X,θ^{old})$. Here is what I am unable to grasp. Why does maximizing the expected value of complete data log likelihood under the posterior distribution w.r.t $θ$ give a better estimate $θ^{new}$?

I can get the intuition of why maximizing the log likelihood(and not the expected value of log likelihood under some distribution) gives the $θ_{max}$ as we know from the maximum likelihood estimation. But why maximizing the expectation of log likelihood under some distribution also give a better estimate of $θ$?

Also, here what I can mathematically see, $\mathcal{Q}(θ, θ^{old}) = \sum\limits_{Z} p(Z|X,θ^{old})\ln p(X,Z|θ)$
I can see that by expanding I get, $\ln p(X,Z|θ) = \ln p(Z|X,θ) + \ln p(X|θ)$ and substituting I get, $\sum\limits_{Z} p(Z|X,θ^{old})\ln p(Z|X,θ) + \sum\limits_{Z} p(Z|X,θ^{old})\ln p(X|θ)$, in which the 2nd term simply becomes $\ln p(X|θ)$ because it is independent of $Z$.
Thus, $\mathcal{Q}(θ, θ^{old}) = \sum\limits_{Z} p(Z|X,θ^{old})\ln p(Z|X,θ) + \ln p(X|θ)$. And when I substitute value of $\ln p(X|θ)$ and $\mathcal{L}(q,\theta)$ and rearranging, I get $\sum\limits_{Z} p(Z|X,θ^{old})\ln p(Z|X,θ) = -( KL(q||p) + const)$. I am not sure how to make sense of this.

I am referring to Section 9.4 of Patter Recognition and Machine Learning by C. Bishop, if that helps.

Answer 1

I think I got the intuition. I understood after reading the Variational inference part of the Approximate Inference chapter in the book and a section in the Wikipedia article of EM algorithm. I have replaced the $\sum$ with $\int$, so this holds for continuous Z as well. Here it goes.
We can write $p(X|θ)$ as $p(X|θ) = \frac{p(X,Z|θ)}{p(Z|X,θ)} = \frac{p(X,Z|θ)/q(Z)}{p(Z|X,θ)/q(Z)} $. Applying log we get, $\ln p(X|θ) = \ln \frac{p(X,Z|θ)}{q(Z)} - \ln\frac{p(Z|X,θ)}{q(Z)} $. Multiplying by $q(Z)$ on both sides and integrating w.r.t to Z we get $$ \ln p(X|θ) \int q(Z)dZ = \int q(Z) \ln \frac{p(X,Z|θ)}{q(Z)} dZ - \int q(Z) \ln\frac{p(Z|X,θ)}{q(Z)}dZ $$ So finally we can write $$ \ln p(X|θ) = \mathcal{L}(q,θ) + KL(q||p) $$ where $$\mathcal{L}(q,θ) = \int q(Z) \ln \frac{p(X,Z|θ)}{q(Z)} dZ $$ $$ KL(q||p) = -\int q(Z) \ln \frac{p(Z|X,θ)}{q(Z)} dZ $$ My intuition says that we want to express this as the familiar concept of lower bound(1st term) and KL divergence(2nd term). Here $q(Z)$ is our approximation of the latent variable posterior distribution and we want to make it as good an approximation as possible. Which means the KL divergence term will become 0 when $q(Z) = p(Z|X,θ)$(best possible). So here minimizing the KL divergence is equal to maximizing lower bound as both of them sum to $\ln p(X|θ)$ which is constant w.r.t Z. On expanding

$$\mathcal{L}(q,θ) = \int q(Z) \ln p(X,Z|θ)dZ - \int q(Z) \ln q(Z) dZ$$ To see how maximizing the expected complete-data log likelihood under the latent variable posterior distribution maximizes $\mathcal{L}(q,θ)$ at least as much, we do the following. We make an initial guess for $q(Z)$ by choosing a random value for $\theta$ and we get $q(Z) = p(Z|X,\theta^{old})$. Putting it in the above equation, we get, $$\mathcal{L}(q,θ) = \int p(Z|X,\theta^{old}) \ln p(X,Z|θ)dZ - \int p(Z|X,\theta^{old}) \ln p(Z|X,\theta^{old}) dZ \\ = \mathbb{E}[\ln p(X,Z)dZ] + const $$ where $const$ is the entropy of $p(Z|X,\theta^{old})$ and is independent of $\theta$. Now maximizing the expectation term w.r.t $\theta$ we get a better estimate of $\mathcal{L}(q)$ and since the KL divergence is non-negative, $\ln p(X)$ increases at least as much as the increase in $\mathcal{L}(q)$.

References:

• Wikipedia - An alternate explanation that really clicked for me.