6
$\begingroup$

I'm working my way through the derivation of EM in Bishop (p. 435).

I'm stuck trying to derive to MLE for $\mu_k$ for the gaussian mixture model.

Basically I get an extra sum in the numerator.

For those that don't have the book:

The log likelihood for the gaussian mixture model is:

$$ ln\; p(X|\pi,\mu,\Sigma) = \sum_{n=1}^{N} ln \left\{ \sum_{k=1}^K \pi_k N(x_n|\mu_k,\Sigma_k) \right\} $$

When I take derivatives wrt $\mu_k$:

  1. recognise that we're dealing with $ln(f(x))$ and the derivative is $ \frac{f'(x)}{f(x)} $

  2. This gives us:

$$ \sum_{n=1}^{N} \frac{1}{\sum_{k=1}^K \pi_k N(x_n|\mu_k,\Sigma_k)} \times \frac{\partial \sum_{k=1}^K \pi_k N(x_n|\mu_k,\Sigma_k) }{\partial \mu_k} $$

  1. Now only have to solve the differential in the right most term:

$$ \frac{ \partial \sum_{k=1}^K \pi_k N(x_n|\mu_k,\Sigma_k) }{\partial \mu_k} = \sum_{k=1}^K -0.5(2\Sigma^{-1}(x-\mu_k)\times \pi_k N(x_n|\mu_k,\Sigma_k) $$

  1. This leaves me with:

$$ \sum_{n=1}^{N} \frac{ \sum_{k=1}^K \pi_k N(x_n|\mu_k,\Sigma_k) }{\sum_{k=1}^K \pi_k N(x_n|\mu_k,\Sigma_k)} \times -0.5(2\Sigma^{-1}(x-\mu_k)) $$

  1. The solution in the book is:

$$ \sum_{n=1}^{N} \frac{ \pi_k N(x_n|\mu_k,\Sigma_k) }{\sum_{j} \pi_j N(x_n|\mu_j,\Sigma_j)} \times 0.5(2\Sigma^{-1}(x-\mu_k)) $$

How is it that

  1. There's no summation in their numerator?

  2. They've changed what they're summing over (k -> j) ?

  3. They have a positive final term, whereas I have a negative?

Thanks

$\endgroup$

1 Answer 1

2
$\begingroup$

To avoid any confusion, the summation index and the index of the $\mu$ that you differentiate with should be different. From the beginning, assume the likelihood is written with index $j$ and you want to differentiate it with $\mu_k$:

$$\frac{\partial \sum_{j=1}^K \pi_j N(x_n|\mu_j,\Sigma_j) }{\partial \mu_k}=\frac{ \pi_k\partial N(x_n|\mu_k,\Sigma_k)}{\partial \mu_k}$$ which explains why the answer doesn't have a summation in the numerator.

You'll have a minus in $(x-\mu_k)$, i.e. differentiating wrt $\mu_k$ gives $-1$, and also another minus in $\exp(-(\ldots))$ expression in normal PDF. They'll cancel out each other.

$\endgroup$
4
  • $\begingroup$ Thanks @gunes. I'm sorry I really don't understand how the RHS follows from the LHS. I also don't understand why the j's are different to the k's. Aren't we still taking about the different clusters? Thanks $\endgroup$
    – RNs_Ghost
    Mar 10, 2020 at 15:34
  • 1
    $\begingroup$ The summation index can be any letter, but we want to differentiate with respect to a specific $k$. In the summation, if $j\neq k$, the derivative will be $0$, so the only term remaining there will be when $j=k$. $\endgroup$
    – gunes
    Mar 10, 2020 at 15:37
  • $\begingroup$ aha, So this is the reason that we don't differentiate the sum: we're not interested in all of the means, just a specific one? $\endgroup$
    – RNs_Ghost
    Mar 10, 2020 at 15:39
  • 1
    $\begingroup$ exactly, e.g. $\mu_1=a,\mu_2=b$, and you'll differentiate wrt $a$. $\endgroup$
    – gunes
    Mar 10, 2020 at 15:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.