0
$\begingroup$

So I was reviewing the E-step for the Gaussian Mixture model on Wikipedia.

And it looks like in the E-step all you really need to compute is the conditional distribution of Z because that is all that the M-step uses. However in the definition of the EM algorithm it states that in the E-step the Q function must be computed (i.e. the expected log likelihood)? Why doesn't it just say the conditional distribution of Z needs to be computed?

EDIT:

I just noticed that it also says in the article "Speaking of an expectation (E) step is a bit of a misnomer. What are calculated in the first step are the fixed, data-dependent parameters of the function Q. Once the parameters of Q are known, it is fully determined and is maximized in the second (M) step of an EM algorithm." So perhaps the answer is that it is just a misnomer i.e. it is not necessary to compute the full expected value.

$\endgroup$
1
$\begingroup$

Because they are not the same quantity. It is not the conditional distribution of $\mathbf{z}$, but rather it is an expectation taken with respect to this distribution. In other words, the quantity $Q(\theta \mid \theta_{\text{old}})$ is the expectation of $$ \log p(\mathbf{x}, \mathbf{z} \mid \theta) \tag{1} $$ taken with respect to $$ p(\mathbf{z} \mid \mathbf{x}, \theta_{\text{old}}), $$ where $\theta \neq \theta_{\text{old}}$.

The process is sort of intuitive. Usually you maximize (1), but you don't have access the hidden variables. So you pick some parameter $\theta_{\text{old}}$, then average out the hidden stuff. This gives you new function to maximize over all $\theta$: $Q(\theta \mid \theta_{\text{old}})$. Once you pick a new parameter, you keep just keep repeating the steps.

$\endgroup$
  • $\begingroup$ Yes I know it is not the same quantity. I am saying that all the M-step needs is P(Z|x,theta_old). So why not just stop there? Why is there the need to compute Q? The wikipedia page itself actually says "These are called the "membership probabilities" which are normally considered the output of the E step" about P(Z|x,theta_old). $\endgroup$ – user1893354 May 7 '19 at 22:42
1
$\begingroup$

Since$$Q(\boldsymbol\theta\mid\boldsymbol\theta^{(t)}) = \Bbb{E}_{\mathbf{Z}\mid\mathbf{X},\boldsymbol\theta^{(t)}}\left[ \log L (\boldsymbol\theta; \mathbf{X},\mathbf{Z}) \right] = \int _\mathcal Z \log L (\boldsymbol\theta; \mathbf{X},\mathbf{z}) p(\mathbf{z}|\mathbf{X},\boldsymbol\theta^{(t)})\text{d}\mathbf{z}$$it is hard to see why, in general, the knowledge of the conditional distribution $ p(\mathbf{z}|\mathbf{X},\boldsymbol\theta)$ should be enough to find the next value $\boldsymbol\theta^{(t)}$ of $\boldsymbol\theta$ in the EM sequence.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.