Mixing probabilities in mixture models using EM Assume we want to estimate the mixing probabilities ($\pi_{k}$) for each member distribution in the mixture model.
We know that $\sum_{m}^{K}\pi_{m}=1$, so we can formulate the optimization problem as a Lagrangian multiplier; the function we want to maximize (w.r.t. $\pi_{k}$) is is given by,
$$ Q = \sum_{l}^{n}\sum_{m}^{k} p(m|l)\text{ log}q(m|l)+\delta(\sum_{k}\pi_{k}-1) $$
where:
$q(k|l) = \pi_{k}g_{k}(x_{l};\theta_{k})$, i.e. the joint probability of selecting mixture component $k$ and the probability of datapoint $x_{l}$ given component distribution $g_{k}$
and
$p(k|l) = \frac{q(k|l)}{\sum_{m}^{k}q(m|l)}$
Setting $\frac{\partial Q}{\partial \pi_{k}}=0$, yields,
$$ \frac{\partial Q}{\partial \pi_{k}} = \sum_{l}^{n}\frac{p(k|l)}{\pi_{k}}+\delta = 0 $$
Now, in order to solve for $\pi_{k}$, we must solve for $\delta$; apparently, $\delta=-n$. This results in,
$$ \pi_{k} = \sum_{l}^{n} \frac{p(k|l)}{n}$$
However, I'm not sure why this is; more specifically, how can it be shown that,
$$ \sum_{l}^{n}\frac{p(k|l)}{\pi_{k}} = n $$
 A: You hand-waved over the derivation of $\delta=-n$, when in fact it comes from the derivative with respect to $\delta$ and is essentially the condition that $\sum\pi_k=1$. If you want to see the reverse, just plug in the definition of $p(k|l)$ into the formula, and use the fact that $\sum g_k$ over its component is 1, because it is a distribution.
A: When doing EM for a GMM, the log-likelihood of the data is given by: $$\ln p(\mathbf{X}, \mathbf{Z} | \boldsymbol{\mu}, \boldsymbol{\Sigma}, \boldsymbol{\pi}) = \sum_{n=1}^N\sum_{k=1}^K z_{nk} \{\ln \pi_k + \ln \mathcal{N}(\mathbf{x}_n | \boldsymbol{\mu}_k, \boldsymbol{\Sigma}_k)\}$$
To solve for the mixing proportions, we can write first write the Lagrangian $$\mathcal{L}(\mathbf{x}, \delta) = \ln p(\mathbf{X}, \mathbf{Z} | \boldsymbol{\mu}, \boldsymbol{\Sigma}, \boldsymbol{\pi}) + \delta(\sum_{k=1}^K\pi_k - 1)$$ and then take the derivative with respect to $\pi_k$. This gives us $$\frac{\partial\mathcal{L}}{\partial\pi_k} = \sum_{n=1}^N\frac{z_{nk}}{\pi_k} + \delta$$
Equating this to zero and solving for $\pi_k$ gives $\pi_k = \frac{-\sum_{n=1}^N z_{nk}}{\delta}$. Summing both sides over $k$ and employing the constraint $\sum_{k=1}^K\pi_k = 1$, we find that $1 = -N/\delta$. Thus, $\delta= -N$.
