Derivation of M-step for pLSA I was looking at section 6 of these notes and trying to understand the derivation of the M-step at the top of page 10. I understood the derivation for the model without background, but I do not understand where the $(1-P(Z_{d,w}=\theta_B \mid d,w))$ terms come from. Intuitively, I understand that they are down-weighting words that are mostly background and not topic-related. But, how do you get the expressions in the M-step from the derivatives?
 A: This derivation may be more understandable than the one in the paper.
The log likelihood of the background model's joint distribution is the following:
$$
\log p(w,d,Z)=\sum_{d,w}n(d,w)\mathbb{I}\left(Z_{w,d}=\theta_{B}\right)\log\left(\lambda_{B}p(w|\theta_{B})\right)+\sum_{d,w,z}n(d,w)\mathbb{I}\left(Z_{w,d}=z\right)\mathbb{I}\left(Z_{w,d}\neq\theta_{B}\right)\log\left(\left(1-\lambda_{B}\right)p(w|z)p(z|d)p(d)\right)
$$
Then take the expectation to get a bound on the marginal log likelihood
$$
\log p(w,d) \ge
\sum_{d,w} n(d,w)E\left[\mathbb{I}\left(Z_{w,d}=\theta_{B}\right)\right]\log\left(\lambda_{B}p(w|\theta_{B})\right)+\sum_{d,w,z} n(d,w) E\left[\mathbb{I}\left(Z_{w,d}=z\right)\mathbb{I}\left(Z_{w,d}\neq\theta_{B}\right)\right]\log\left(\left(1-\lambda_{B}\right)p(w|z)p(z|d)p(d)\right)
$$
Then take the derivative with respect to the bound to maximize ($\beta$ is a Lagrange multiplier enforcing the normalization constraint $\sum_w p(w|z)=1$):
$$
\frac{\partial H}{\partial p(w|z)}=\frac{\sum_d n(d,w)E\left[\mathbb{I}\left(Z_{w,d}=z\right)\mathbb{I}\left(Z_{w,d}\neq\theta_{B}\right)\right]}{p(w|z)}-\beta=0
$$
$$
p(w|z)\propto \sum_d n(d,w)E\left[\mathbb{I}\left(Z_{w,d}=z\right)\mathbb{I}\left(Z_{w,d}\neq\theta_{B}\right)\right]
$$
The expectation above ends up being
$$
E\left[\mathbb{I}\left(Z_{w,d}=z\right)\mathbb{I}\left(Z_{w,d}\neq\theta_{B}\right)\right]=E\left[\mathbb{I}\left(Z_{w,d}=z\right)\right] E\left[\mathbb{I}\left(Z_{w,d}\neq\theta_{B}\right)\right]=P\left(Z_{w,d}=z\right |w,d)P\left(Z_{w,d}\neq\theta_{B}|w,d\right)=P\left(Z_{w,d}=z\right|w,d) (1-P\left(Z_{w,d}=\theta_{B}|w,d\right))
$$
