I'm using Expectation Maximization algorithm to determine the parameters of Gaussian distributions in a mixture. To get a better understanding of the algorithm, I executed it manually step by step on a small example, following this procedure:
The problem is that in the second iteration log-likelihood decreased.
These are my data: $$X = \begin{bmatrix}1 \\ 2\\ 3\end{bmatrix}$$ The initial means, variances and mixing coefficients are set to:
$$\mu = \begin{bmatrix}1 & 3\end{bmatrix}\quad \pi = \begin{bmatrix}0.5 & 0.5\end{bmatrix}\quad \Sigma=\begin{bmatrix}1 & 1\end{bmatrix}$$
So, I first calculated the initial log-likelihood:
$$L_0 = -4.3893$$
Then I did the E step and calculated $\gamma(z_{nk})$, which I'll shorten as $\gamma_{n,k}$, like this $\gamma_{n,k}=\pi_kN(x_n|\mu_k,\Sigma_k)$, and got:
$$\Gamma = \begin{bmatrix}0.881 & 0.119 \\ 0.5 & 0.5 \\ 0.119 & 0.881\end{bmatrix}$$
Then, I peformed the M step: $$N_1 = 1.5\quad N_2= 1.5$$ $$\pi = \begin{bmatrix}0.5 & 0.5\end{bmatrix}\quad \mu^{new}=\begin{bmatrix}1.492 & 2.508\end{bmatrix}\quad \Sigma=\begin{bmatrix}0.409 & 0.409\end{bmatrix}$$ The new log-likelihood was $L_1=-3.6766$, which was greater than $L_0$.
Then, I did another iteration. In the E step I got: $$\Gamma=\begin{bmatrix}0.9979 & 0.0021 \\ 0.5 & 0.5 \\ 0.0021 & 0.9979 \end{bmatrix}$$ In the M step I got: $$N_1=1.5\quad N_2=1.5$$ $$\pi = \begin{bmatrix}0.5 & 0.5\end{bmatrix}\quad \mu^{new}=\begin{bmatrix}1.3361 & 2.6639\end{bmatrix}\quad \Sigma=\begin{bmatrix}0.2259 & 0.2259\end{bmatrix}$$ for which the log-likelihood was $L_2=-6.2123$.
However, $L_2 < L_1$! Even though I programmed the algorithm in python and got the same results, I know that $L_i$ ($i=0,1,2,\ldots)$ should be monotonically increasing, so, there has to be some mistake, but I can't catch it. Can anyone help?