# Does the sum of updated mixture weights in EM algorithm equal 1 in M step, why?

I am usign EM algorithm to estimate my model's parameters. As you know, the mixture weight must be sum to one. $\sum \pi_n = 1$ where $n$ is the number of mixture component. In M step we can find the updated mixture weight for each iteration as follows: $\pi^{i+1} = mean(\tau)$ where $i$ is the number of iteration. My question here, how to prove that, the sum of updated mixture weight will also equal to one?

Let us say that we consider a mixture model of $K$ components and the inner density is $f(\dot;\Theta)$, i.e. a gaussian density if we consider GMMs and so on.

First of all the Q-function looks like

$$Q(\Theta, \Theta^{(\text{old})}) = \sum_{i=1}^n \sum_{k=1}^K T_{i,k}^{(\text{old})} \left[ \log \tau_k + \log f(x_i;\Theta)\right]$$

(see https://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm for the GMM case with $k=2$ but it works just as fine with any other density $f(\dot;\Theta)$ and $K$ arbitrary).

It is easy to see that whenever you have two functions $F, G$ then $$\text{argmax}_{\Theta} (F(\Theta) + G(\Theta)) = \text{argmax}_{\Theta} F(\Theta) + \text{argmax}_{\Theta} G(\Theta)$$

so that we can maximize the $\tau$ and the $\Theta$ individually. Now we want to get

$$\text{argmax}_{\tau} \sum_{k=1}^K \left(\underbrace{\sum_{i=1}^n T_{i,k}^{(\text{old})}}_{=: A_k} \right) \log(\tau_k)$$

under the condition that $\sum_{k=1}^K \tau_k = 1$ and $0 < \tau_k < 1$, i.e. the fact that $\sum_k \tau_k = 1$ does not pop out magically but rather we demand it and we maintain it throughout the execution of the EM algorithm and we have to work for it.

Put $f(\tau) = \sum_{k=1}^K A_k \log(\tau_k)$. Now, solving this problem is just the right application for the theorem of Lagrange (https://en.wikipedia.org/wiki/Lagrange_multiplier), i.e. we put

$$g(\tau) = (\sum_{k=1}^K \tau_k) - 1$$

and now we know that any maximal $\tau$ must be a 'critical' point of the so-called Lagrange function

$$L(\tau, \lambda) = f(\tau) - \lambda g(\tau)$$

i.e. we differentiate by each $\tau_k$ and by $\lambda$ and all these have to be zero. Differentiating by $\tau_k$ (and setting to zero) yields

$$0 = \partial_{\tau_k} L = \frac{A_k}{\tau_k} - \lambda$$ i.e. $$\tau_k = -\frac{A_k}{\lambda}$$ differentiating by $\lambda$ and setting to zero yields the (original) restriction $$0 = \partial_{\lambda} L = g(\tau) = (\sum_{k=1}^K \tau_k) - 1$$

so that

$$1 = \sum_{k=1}^K \tau_k = -\sum_{k=1}^K \frac{A_k}{\lambda} = \frac{-\sum_{k=1}^K A_k}{\lambda}$$

thus

$$\lambda = -\sum_{k=1}^K A_k$$

and

$$\tau_k = -\frac{A_k}{\lambda} = \frac{A_k}{\sum_{k'=1}^K A_{k'}}$$

Now it is easy to see that $\sum_{k} \tau_k = 1$ but this is not a suprise!!! We created the $\tau_k$ in a way that this equation must be satisfied.

• Thank you for your very good answer. I was thinking about that today. However, if someone asked why "Now it is easy to see that $∑_{k}τ_{k}=1$"? is the answer because we take the mean of the $A_k$? sorry, if this is a silly question. Commented May 10, 2017 at 15:55
• Just sum them up then you arrive at $\sum_{k} A_k / \sum_{k'} A_{k'}$ and then the denominator and the numerator are the same. Just replace k' by k... Commented May 10, 2017 at 16:05