EM algorithm gaussian mixtures- derivation

Question

I'm trying to make sense of a derivation I'm following from the lecture notes of Stanford's ML course. Specifically the notes are here: http://cs229.stanford.edu/notes/cs229-notes8.pdf

I'm particular I'm interested in the step where they are optimizing the lower bound of the log-likelihood by finding the derivative of that function w.r.t the mean (top of page 7). I'm a bit stuck on going from line 1 to line 2 in that sequence. To copy here (using abbreviated notation for space):

$$ \nabla \mu_s \sum_{i=1}^{m}\sum_{j=1}^{k} w_j^{(i)}\log\frac{\alpha_j \exp\psi_{j}^{(i)}\phi_j}{w_j^{(i)}} \qquad(1) $$ $$ \nabla \mu_s \sum_{i=1}^{m}\sum_{j=1}^{k} w_j^{(i)}\psi_{j}^{(i)}\qquad(2) $$ where $\psi_{j}^{(i)} =-\frac{1}{2}(x^{(i)}-\mu_j)^T\Sigma^{-1}_{j}(x^{(i)}-\mu_j)$ and $\alpha_j=\frac{1}{(2\pi)^{n/2}|\Sigma_j|^{1/2}}$. The quantity $\mu_s$ is the mean (of a multivariate gaussian distribution) for class $s$.

In short, I'm not sure how they get from (1) to (2). From just changing the log of (1) I see:

$$ \nabla \mu_s \sum_{i=1}^{m}\sum_{j=1}^{k} w_j^{(i)}\log \left(\alpha_j \exp\psi_{j}^{(i)}\phi_j\right) - w_j^{(i)}\log w_j^{(i)} \qquad(1) $$

To get to their form (2) my impression is that they take pre-emptively discard terms that would contain derivatives of $w_j^{(i)}$ (w.r.t. $\mu_s$). At least if you assume that much, you get to (2). However, as they show earlier in that PDF,

$$ w_j^{(i)} = p(z^{(i)}=j | x^{(i)}) =\frac{p(x^{(i)}|z^{(i)}=j)p(z^{(i)}=j) }{\sum_j p(x^{(i)},z^{(i)}=j) } $$ i.e. this quantity is the posterior distribution of the class labels. Intuitively, it seems like $\partial w_j^{(i)}/ \partial \mu_s$ should be non-zero. After all, changing the location of the mean $\mu_s$ would change your posterior probability for a particular class given the same data. (Right?). Just from a mechanical point of view, $\partial w_j^{(i)}/\partial \mu_j$ would seemingly have some non-zero terms since $p(x^{(i)}|z^{(i)}=j) \sim \mathcal{N}(\mu_j, \Sigma_j)$. If I work through the algebra, I get

$$ \frac{\partial w_j^{(i)}}{\partial \mu_j} = w_j^{(i)}(1-w_j^{(i)})\frac{\partial \psi^{(i)}_{j}}{\partial \mu_j} $$

What am I missing here? If I take the log-likelihood (without even thinking about EM algorithm),

$$ \ell = \sum_{i=1}^{m}\log \sum_{j=1}^{k}p(x^{(i)}|z^{(i)}=j)p(z^{(i)}=j) $$ and differentiate w.r.t $\mu_s$, I correctly get

$$ \mu_s^* = \frac{\sum_{i=1}^{m}w_s^{(i)}x^{(i)}}{\sum_{i=1}^{m}w_s^{(i)}} $$

so I'm comfortable with the eventual result, just not the intermediate steps.

Answer 1

The most difficult part in the EM algorithm [in my teaching experience] is the co-existence of two sets of parameters $\theta$, one that is the "current" value of the parameter, $\theta^{(t)}$, obtained at the previous iteration $t$, and one that is free [and free to be optimised], both within the target "E" function $$Q(\theta;\theta^{(t)}) = \mathbb{E}_{\theta^{(t)}}\left[\log L^c(\theta|X,Z) | X=x \right]$$ which is more correctly a function of $\theta$ indexed by $\theta^{(t)}$.

In the case of a mixture model, this $Q(\theta;\theta^{(t)})$ function writes down as $$\sum_{i=1}^n \sum_{j=1}^k \mathbb{P}_{\theta^{(t)}}\left[Z_i=j|X_i=x_i\right]\left\{\log(w_j)+ \log(f(x_i|\zeta_j) \right\}$$ where $\zeta_j$ denotes the parameter(s) of the $j$th component of the mixture and $w_j$ the (known or unknown) weight of this component. Thus $$\theta=(\zeta_1,\ldots,\zeta_k,w_1,\ldots,w_k)$$In this expression, the conditional probability only depends on the "current" value of the parameter, $\theta^{(t)}$, hence is not to be differentiated in terms of the free parameter.