2
$\begingroup$

I am having a hard time figuring out the equation 9.39 page 443 in Bishop's book : Pattern recognition and Machine Learning.

A posterior distribution in the book is written as: $$ p(\textbf{Z} \mid \textbf{X}, \mu, \pi) \approx \Pi_n \Pi_k [\pi_k \mathcal{N}(x_n \mid \mu_k, \Sigma_k)]^{z_{nk}} $$

where $z$ is a 1-of-K binary rperesentation in which a particular element $z_k \in \{0,1 \}$ and $\sum_k z_k=1$ and $z_{nk}$ denotes the indicator for the $n$th point.

Then the author states that: $$ \mathbb{E}[z_{nk}] = \frac{\sum_{z_{nk}} z_{nk} [\pi_k \mathcal{N}(x_n \mid \mu_k, \Sigma_k)]^{z_{nk}}}{\sum_{z_{nj}} [\pi_j \mathcal{N}(x_n \mid \mu_j, \Sigma_j)]^{z_{nj}}} $$

I don't really understand where this expression comes from. This is not the classical definition of the expectation of a random variable.

$\endgroup$

1 Answer 1

2
$\begingroup$

Since $$ p(Z\mid X,\mu,\Sigma,\pi)\propto \prod_{n=1}^N \prod_{k=1}^K \left\{\pi_k\cdot\mathrm{N}(x_n\mid \mu_k,\Sigma_k)\right\}^{z_{nk}} $$ factors over $n$, the $z_n$'s are conditionally independent and $$ p(z_n\mid x_n,\mu,\Sigma,\pi)\propto \prod_{k=1}^K \left\{\pi_k\cdot\mathrm{N}(x_n\mid \mu_k,\Sigma_k)\right\}^{z_{nk}}. $$ Now, $z_{nk}=1$ if and only if $z_{n\ell}=0$ for $\ell\neq k$. Hence, $$ p(z_{nk} = 1\mid x_n,\mu,\Sigma,\pi) =\frac{\pi_k\cdot\mathrm{N}(x_n\mid \mu_k,\Sigma_k)}{\sum_{\ell=1}^K \left\{\pi_\ell\cdot\mathrm{N}(x_n\mid \mu_\ell,\Sigma_\ell)\right\}^\ell} = (*). $$ Since $z_{nk}$ is an indicator, the expectation $$ \mathbb{E}[z_{nk}\mid x_n,\mu,\Sigma,\pi]=(*). $$

$\endgroup$
2
  • $\begingroup$ I am not sure I understand before the last line and the last line. It is because if I expand the expectation formula $\mathbb{E}[z_{nk}] = 1 * p(z_{nk}=1 \mid x_n, \mu, \Sigma, \pi) + 0 * p(z_{nk}=0\mid x_n, \mu, \Sigma, \pi)$ ? $\endgroup$
    – glouis
    Feb 5, 2020 at 15:19
  • $\begingroup$ Exactly. Perfect. $\endgroup$
    – Zen
    Feb 5, 2020 at 22:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.