0
$\begingroup$

Exercise 9.11 from Bishop's "Pattern Recognition and Machine Learning" asks

Show that in the limit $\epsilon\to0$, maximizing the expected complete-data log likelihood for the Gaussian mixture model in which all components have covariance $\epsilon I$, given by (9.40), $$ \mathbb{E}_Z[\ln p(X,Z|\mu,\Sigma,\pi)] = \sum_n \sum_k \gamma(z_{nk}) \bigl\{ \ln\pi_k + \ln\mathcal{N}(x_n|\mu_k,\Sigma_k) \bigr\} \tag{9.40} $$ is equivalent to minimizing the distortion measure $J$ for the $K$-means algorithm given by (9.1) $$ J = \sum_n \sum_k r_{nk} \|x_n-\mu_k\|^2. \tag{9.1} $$

I agree that $\gamma(z_{nk})\to r_{nk}$ as $\epsilon\to0$. But I could not derive the equation (9.43) $$ \mathbb{E}_Z[\ln p(X,Z|\mu,\Sigma,\pi)] \to -\frac{1}{2} \sum_n \sum_k r_{nk} \|x_n-\mu_k\|^2 + \text{const} \tag{9.43} $$ as $\epsilon\to0$. Any comment is welcome.

I leave an answer here for a reference in future.

Answer: (with the help of KDG)

The equation (9.43) in the book is not correct. The correct one is $$ \epsilon\cdot\mathbb{E}_Z[\ln p(X,Z|\mu,\Sigma,\pi)] \to -\frac{1}{2} \sum_n \sum_k r_{nk} \|x_n-\mu_k\|^2 $$ as $\epsilon\to0$. (See stats.stackexchange.com/a/129918/212274 for further discussion.)

Improve this question
$\endgroup$
  • 1
    $\begingroup$ Is there any possibility if the definition of distance is $\| x_n-\mu_k\|^2=( x_n-\mu_k)'\Sigma^{-1}( x_n-\mu_k)$ as Mahalanobis distance? If it is. Just putting the pdf of multivariate normal distribution $\mathcal{N}(x_n|\mu_k,\Sigma_k) $ would lead to the equation 9.43 because $-1/2 \| x_n-\mu_k\|^2$ is the exponent of the pdf. I am not good at Machine learning so I could be wrong. $\endgroup$ – KDG Aug 29 '18 at 6:11
  • $\begingroup$ @KDG Thanks for your comment. The definition of distance is surely $\|x_n-\mu_k\|^2=(x_n-\mu_k)^T(x_n-\mu_k)$. Actually the problem is that $-1/(2\epsilon)\|x_n-\mu_k\|^2$ is the exponent of the pdf because the covariance matrix $\Sigma_k=\epsilon I$, where $I$ is the identity matrix. $\endgroup$ – ChoF Aug 29 '18 at 11:16
  • 1
    $\begingroup$ The problem is that $\epsilon$ is missing, right? May I recommend you to check out this link? : stats.stackexchange.com/a/129918/212274. Maybe it's kind of typo. I guess the equation should be $\epsilon \mathbb{E}_Z[\ln p(X,Z|\mu,\Sigma,\pi)] \to -\frac{1}{2} \sum_n \sum_k r_{nk} \|x_n-\mu_k\|^2 + \text{const}$ which has $\epsilon$ at first. $\endgroup$ – KDG Aug 29 '18 at 12:27
  • $\begingroup$ @KDG Thank you very much for the link. That's exactly what I want to find. I agree your answer that the expectation in (9.43) must have $\epsilon$ at first. Then I could finish here with the answer that the equation (9.43) is not correct. $\endgroup$ – ChoF Aug 29 '18 at 14:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.