M step EM algorithm in Mixture Models. Expected value of the indicator variable under the posterior [closed]

I am not able to solve the following expectation. In the EM algorithm, the first step in the M step is to compute the expected value of $$\log p(x,z)$$ where $$x$$ are observations and $$z$$ indicator variables. Basically we end up having to solve:

$$\mathbb{E}_{p(z|X)}\log p(x,z)=\underset{k}{\sum}\underset{z_k}{\sum}z_k p(z_k|X)$$

I am interested in showing that:

$$\underset{z}{\sum}z_k p(z_k|X)= \underset{z}{\sum}z_k [\pi_k\mathcal{N}(x|\mu_k,C_k)]^{z_k} = \pi_k\mathcal{N}(x|\mu_k,C_k) =p(z_k|X)$$

I guess that the summation over $$z$$ implies summing over all the $$K$$ possible one-hot vectors $$z$$. If I apply this I don't get the result.

• In the expectation step, we just get the lower bound function of the parameters using the current temporary parameters to calculate the z's, and z's are not (hard) one hot(which is for k-means) but (soft) multinomial. I wonder where the derivation you are interested in is coming from? Commented May 7, 2020 at 16:55
• The words one-hot vector seem a bit jargonesque for $n$-space orthogonality, Perhaps that is contributing to an impression of a lack of clarity. I upvoted this and @Xi'an 's answer, but it may wind up being closed despite that. Perhaps consider editing a bit for clarity as I do not see anything amiss in the question, but broader accessibility, such as might be provided by defining variable names might improve the quality of attention the question seems to be attracting.
– Carl
Commented May 14, 2020 at 7:10
• Without further detail, the first equation is incorrect, missing a $\log$ on the rhs. Commented May 14, 2020 at 7:56

A typo: the $$\log$$ is missing from the rhs: $$\mathbb{E}_{p(\cdot|X)}[\log p(X,Z)|X]=\underset{k}{\sum}\sum_{z_k\in\mathcal Z}p(z_k|X_k) \log p(z_k|X_k)p(X_k)$$
Furthermore, given that the EM algorithm involves two types of parameters, the current one, $$\theta^0$$ say, and the free one, $$\theta$$ say, it would be safer to write the conditional expectation as $$\mathbb{E}_{p(\cdot|\theta^0,X)}[\log p(X,Z|\theta)|X]=\underset{k}{\sum}\underset{z_k\in\mathcal Z}{\sum} p(z_k|\theta^0,X_k) \log \{p(z_k|X_k,\theta)p(X_k|\theta)\}$$
Although the question does not mention it, this model seems to be a full Gaussian mixture model $$p(x_k)=\sum_{i=1}^I \pi_i \varphi(x_k;\mu_i,C_i)$$ for which $$\underset{k}{\sum}\underset{z_k\in\mathcal Z}{\sum} p(z_k|\theta^0,X_k) \log \{p(z_k|X_k,\theta)p(X_k|\theta)\}$$ is equal to $$\underset{k}{\sum}\underset{z_k\in\mathcal Z}{\sum} \frac{\pi^0_{z_k}\varphi(x_k;\mu^0_{z_k},C_{z_k}^0)}{\sum_{i=1}^I \pi_i^0 \varphi(x_k;\mu^0_i,C_i^0)}\,\log\left\{ \pi_{z_k} \varphi(x_k;\mu_{z_k},C_{z_k})\right\}$$ If $$z_k$$ ($$k$$ being the index of the $$k$$-th observation $$x_k$$) is a vector of binary indicators, $$z_k=(z_{k1},\ldots,z_{kI})\in\{0,1\}^I$$ then $$p(x_k,z_k|\theta)=\prod_{i=1}^I \pi_i^{z_{ki}} \varphi(x_k;\mu_i,C_i)^{z_{ki}}$$ meaning $$\log p(x_k,z_k|\theta)=\sum_{i=1}^I z_{ki}\left\{\log\pi_i +\log\varphi(x_k;\mu_i,C_i)\right\}$$ and $$\mathbb{E}_{p(\cdot|\theta^0,X)}[\log p(X,Z|\theta)|X]= \sum_k \sum_{i=1}^I \mathbb{E}_{p(\cdot|\theta^0,X)}[z_{ki}|X] \left\{\log\pi_i+\log\varphi(x_k;\mu_i,C_i)\right\}$$
• sorry. I assumed that I was using the standard notation for this model, in which $z$ is one-hot encoded. Anyway, I haven't asked for the expectation of the joint, but the expectation of $z$ under the posterior. Commented May 7, 2020 at 17:32
• well, for the mixture model it is. Anyway, my question was the expectation of $z_k$ under the posterior, not the expectation of the joint or the $\log$ joint. I just say that to compute the expectation of the log of the joint we end up having to compute the expectation of $z$ under the posterior, and that is my question. Thanks anyway. Commented May 8, 2020 at 15:12