I'm working my way through the derivation of EM in Bishop (p. 435).

I'm stuck trying to derive to MLE for $\mu_k$ for the gaussian mixture model.

Basically I get an extra sum in the numerator.

For those that don't have the book:

The log likelihood for the gaussian mixture model is:

$$ ln\; p(X|\pi,\mu,\Sigma) = \sum_{n=1}^{N} ln \left\{ \sum_{k=1}^K \pi_k N(x_n|\mu_k,\Sigma_k) \right\} $$

When I take derivatives wrt $\mu_k$:

  1. recognise that we're dealing with $ln(f(x))$ and the derivative is $ \frac{f'(x)}{f(x)} $

  2. This gives us:

$$ \sum_{n=1}^{N} \frac{1}{\sum_{k=1}^K \pi_k N(x_n|\mu_k,\Sigma_k)} \times \frac{\partial \sum_{k=1}^K \pi_k N(x_n|\mu_k,\Sigma_k) }{\partial \mu_k} $$

  1. Now only have to solve the differential in the right most term:

$$ \frac{ \partial \sum_{k=1}^K \pi_k N(x_n|\mu_k,\Sigma_k) }{\partial \mu_k} = \sum_{k=1}^K -0.5(2\Sigma^{-1}(x-\mu_k)\times \pi_k N(x_n|\mu_k,\Sigma_k) $$

  1. This leaves me with:

$$ \sum_{n=1}^{N} \frac{ \sum_{k=1}^K \pi_k N(x_n|\mu_k,\Sigma_k) }{\sum_{k=1}^K \pi_k N(x_n|\mu_k,\Sigma_k)} \times -0.5(2\Sigma^{-1}(x-\mu_k)) $$

  1. The solution in the book is:

$$ \sum_{n=1}^{N} \frac{ \pi_k N(x_n|\mu_k,\Sigma_k) }{\sum_{j} \pi_j N(x_n|\mu_j,\Sigma_j)} \times 0.5(2\Sigma^{-1}(x-\mu_k)) $$

How is it that

  1. There's no summation in their numerator?

  2. They've changed what they're summing over (k -> j) ?

  3. They have a positive final term, whereas I have a negative?



To avoid any confusion, the summation index and the index of the $\mu$ that you differentiate with should be different. From the beginning, assume the likelihood is written with index $j$ and you want to differentiate it with $\mu_k$:

$$\frac{\partial \sum_{j=1}^K \pi_j N(x_n|\mu_j,\Sigma_j) }{\partial \mu_k}=\frac{ \pi_k\partial N(x_n|\mu_k,\Sigma_k)}{\partial \mu_k}$$ which explains why the answer doesn't have a summation in the numerator.

You'll have a minus in $(x-\mu_k)$, i.e. differentiating wrt $\mu_k$ gives $-1$, and also another minus in $\exp(-(\ldots))$ expression in normal PDF. They'll cancel out each other.

  • $\begingroup$ Thanks @gunes. I'm sorry I really don't understand how the RHS follows from the LHS. I also don't understand why the j's are different to the k's. Aren't we still taking about the different clusters? Thanks $\endgroup$ – RNs_Ghost Mar 10 '20 at 15:34
  • 1
    $\begingroup$ The summation index can be any letter, but we want to differentiate with respect to a specific $k$. In the summation, if $j\neq k$, the derivative will be $0$, so the only term remaining there will be when $j=k$. $\endgroup$ – gunes Mar 10 '20 at 15:37
  • $\begingroup$ aha, So this is the reason that we don't differentiate the sum: we're not interested in all of the means, just a specific one? $\endgroup$ – RNs_Ghost Mar 10 '20 at 15:39
  • 1
    $\begingroup$ exactly, e.g. $\mu_1=a,\mu_2=b$, and you'll differentiate wrt $a$. $\endgroup$ – gunes Mar 10 '20 at 15:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.