Unclear part of EM algorithm steps I am learning the EM algorithm. As I understand the steps are as follows:
1- initialise the parameters. 
2- Then start with E-step
3- maximize the log-likelihood function (from E step) and find the new estimate of the parameters. 
4-iterate till the difference between the last and new log likelihood is small. 
My question stem from em in r on this site. I do not understand why at the first they write log likelihood as this:
loglik[2]<-mysum(pi1*(log(pi1)+log(dnorm(dat,mu1,sigma1))))+mysum(pi2*(log(pi2)+log(dnorm(dat,mu2,sigma2))))
Why do they multiply pi1 twice pi1*(log(pi1).
 A: I agree with you, this is not the actual formula of the log likelihood. What they should have implemented is
$$ \sum_{j} z_{1,j} [\log(\pi_1) + \log(f(x_j;\Psi^\text{old}_1))] + \sum_{j} z_{2,j} [\log(\pi_2) + \log(f(x_j;\Psi^\text{old}_2))]$$
where $j$ runs over the training samples indices. What they have implemented seems indeed to be
$$ \sum_{j} \pi_1 [\log(\pi_1) + \log(f(x_j;\Psi^\text{old}_1))] + \sum_{j} \pi_2 [\log(\pi_2) + \log(f(x_j;\Psi^\text{old}_2))]$$
HOWEVER this does not make much of a difference: in a more formal language, the $z_{i,j}$ are the conditional densities/probabilities $f_{Z_i|X_j}(i|x_j)$ i.e. something interpretable as the probability that $x_j$ ''comes from
'' the $i$-th density in the mixture. Now notice that they only do this weird 'wrong' multiplication in the very beginning... in the loop the seem to have implemented the exact formula above. Phrased differently: If you have no prior knowledge and you have two components, what is the initial probability that some random training sample comes from the first density? You have to specify some value in order to make the next step and improve you prior believe... So, what value do you set initially? Well, since you do not want to put in any prior knowledge it would be a good idea to put $z_{i,j} = 1/\text{amount components}$, i.e. $z_{i,j} = 1/2$ in the case of two densities. By 'accident', this is exactly the initial value that they have assigned to pi1 (well, up to the minus sign... this is a mere question of whether you want to maximize $\log L$ or minimize $-\log L$). Hence, they should have written
loglik[2]<-mysum( -(1/2) * (log(pi1) + ...
but the author has decided to write
loglik[2]<-mysum( pi1 * (log(pi1) + ...
which is confusing (because philosophically it is abolutely wrong, the pi does not have to go in there) but in the end, the evaluated value will be exactly the same...
Does that make sense?
