0
$\begingroup$

Assume we want to estimate the mixing probabilities ($\pi_{k}$) for each member distribution in the mixture model.

We know that $\sum_{m}^{K}\pi_{m}=1$, so we can formulate the optimization problem as a Lagrangian multiplier; the function we want to maximize (w.r.t. $\pi_{k}$) is is given by,

$$ Q = \sum_{l}^{n}\sum_{m}^{k} p(m|l)\text{ log}q(m|l)+\delta(\sum_{k}\pi_{k}-1) $$

where:

$q(k|l) = \pi_{k}g_{k}(x_{l};\theta_{k})$, i.e. the joint probability of selecting mixture component $k$ and the probability of datapoint $x_{l}$ given component distribution $g_{k}$

and

$p(k|l) = \frac{q(k|l)}{\sum_{m}^{k}q(m|l)}$

Setting $\frac{\partial Q}{\partial \pi_{k}}=0$, yields,

$$ \frac{\partial Q}{\partial \pi_{k}} = \sum_{l}^{n}\frac{p(k|l)}{\pi_{k}}+\delta = 0 $$

Now, in order to solve for $\pi_{k}$, we must solve for $\delta$; apparently, $\delta=-n$. This results in,

$$ \pi_{k} = \sum_{l}^{n} \frac{p(k|l)}{n}$$

However, I'm not sure why this is; more specifically, how can it be shown that,

$$ \sum_{l}^{n}\frac{p(k|l)}{\pi_{k}} = n $$

$\endgroup$
2
$\begingroup$

You hand-waved over the derivation of $\delta=-n$, when in fact it comes from the derivative with respect to $\delta$ and is essentially the condition that $\sum\pi_k=1$. If you want to see the reverse, just plug in the definition of $p(k|l)$ into the formula, and use the fact that $\sum g_k$ over its component is 1, because it is a distribution.

$\endgroup$
0
$\begingroup$

When doing EM for a GMM, the log-likelihood of the data is given by: $$\ln p(\mathbf{X}, \mathbf{Z} | \boldsymbol{\mu}, \boldsymbol{\Sigma}, \boldsymbol{\pi}) = \sum_{n=1}^N\sum_{k=1}^K z_{nk} \{\ln \pi_k + \ln \mathcal{N}(\mathbf{x}_n | \boldsymbol{\mu}_k, \boldsymbol{\Sigma}_k)\}$$ To solve for the mixing proportions, we can write first write the Lagrangian $$\mathcal{L}(\mathbf{x}, \delta) = \ln p(\mathbf{X}, \mathbf{Z} | \boldsymbol{\mu}, \boldsymbol{\Sigma}, \boldsymbol{\pi}) + \delta(\sum_{k=1}^K\pi_k - 1)$$ and then take the derivative with respect to $\pi_k$. This gives us $$\frac{\partial\mathcal{L}}{\partial\pi_k} = \sum_{n=1}^N\frac{z_{nk}}{\pi_k} + \delta$$ Equating this to zero and solving for $\pi_k$ gives $\pi_k = \frac{-\sum_{n=1}^N z_{nk}}{\delta}$. Summing both sides over $k$ and employing the constraint $\sum_{k=1}^K\pi_k = 1$, we find that $1 = -N/\delta$. Thus, $\delta= -N$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.