enter image description here

In Bishop's Pattern Recognition and Machine Learning book(page440), it talks about the M-step in EM algorithm of Gaussian Mixture Model. I am confused about the likelihood function of M-step. By likelihood, it is usually $p(X|\theta)$, right? But when involving this latent variable in the maximum likelihood function, it becomes a bit hard to understand. In the last sentence of the first paragraph, it says: "we shall suppose that maximization of this complete-data log likelihood function is traightforward", then in the second paragraph, it writes:" Because we cannot use the complete-data log likelihood, we consider...". Why cannot we use the complete-data log likelihood if it is straightforward to maximize? Why taking the expected value under the posterior distribution of the latent variable makes sense? What is the issue if we just maximize $p(X,Z|\theta)=p(Z)p(X|Z,\theta)=\prod_{n=1}^{N}\prod_{k=1}^K{\pi_k^{z^{nk}}}{N(x_n|\mu_k,\Sigma_k})^{z_{nk}}$


1 Answer 1


you might consider rewording your question a little bit as I find the question itself confusing. Nevertheless, let me try to re-word what Bishop has said in more plain language.

Firstly as regards we shall suppose that maximisation of this complete-data log likelihood function is straightforward

He means that if we had access to both X and Z, then it would be straightforward to find the $\theta$ which maximises this log-likelihood.

We don't have the values of Z, i.e. we don't know $Z_{1}=a, X_{2}=b, \ldots$ But we do have a posterior over all the values of Z, which he calls $P(Z|X, \theta)$

So while we can't maximise $P(X,Z|\theta)$ wrt $\theta$, we can maximise

$\langle P(X,Z|\theta) \rangle_{z}$

wrt $\theta$

This expectation over Z integrates out all z-dependence, the expression will depend on just $(X,\theta)$ and you know X, it's your observable data, so now you have something which is a function of $\theta$ which you wish to maximise wrt $\theta$

Note that the expectation wrt z above means integrating wrt the posterior, explicitly this is given by

$\int P(X,Z|\theta) P(Z|X,\theta _{t-1})dZ $

The somewhat non-obvious point is that I've written $\theta$ and $\theta _{t-1}$. Whatever value $\theta$ took at timestep t-1 gets subbed into that integral, and then you maximise the integral wrt $\theta$ to calculate $\theta_{t}$

Bishop then goes on to prove that this somewhat non-obvious procedure (unlike gradient descent) guarantees your log-likelihood will increase at every step

(and while I've written the above in terms of maximising $\langle P(X,Z|\theta) \rangle_{z}$ wrt $\theta$ for notational simplicity, in practice you maximise its logarithm wrt $\theta$)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.