I was looking at section 6 of these notes and trying to understand the derivation of the M-step at the top of page 10. I understood the derivation for the model without background, but I do not understand where the $(1-P(Z_{d,w}=\theta_B \mid d,w))$ terms come from. Intuitively, I understand that they are down-weighting words that are mostly background and not topic-related. But, how do you get the expressions in the M-step from the derivatives?
Tell me more
×
Cross Validated is a question and answer site for
statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
|
This derivation may be more understandable than the one in the paper. The log likelihood of the background model's joint distribution is the following: $$ \log p(w,d,Z)=\sum_{d,w}n(d,w)\mathbb{I}\left(Z_{w,d}=\theta_{B}\right)\log\left(\lambda_{B}p(w|\theta_{B})\right)+\sum_{d,w,z}n(d,w)\mathbb{I}\left(Z_{w,d}=z\right)\mathbb{I}\left(Z_{w,d}\neq\theta_{B}\right)\log\left(\left(1-\lambda_{B}\right)p(w|z)p(z|d)p(d)\right) $$ Then take the expectation to get a bound on the marginal log likelihood $$ \log p(w,d) \ge \sum_{d,w} n(d,w)E\left[\mathbb{I}\left(Z_{w,d}=\theta_{B}\right)\right]\log\left(\lambda_{B}p(w|\theta_{B})\right)+\sum_{d,w,z} n(d,w) E\left[\mathbb{I}\left(Z_{w,d}=z\right)\mathbb{I}\left(Z_{w,d}\neq\theta_{B}\right)\right]\log\left(\left(1-\lambda_{B}\right)p(w|z)p(z|d)p(d)\right) $$ Then take the derivative with respect to the bound to maximize ($\beta$ is a Lagrange multiplier enforcing the normalization constraint $\sum_w p(w|z)=1$): $$ \frac{\partial H}{\partial p(w|z)}=\frac{\sum_d n(d,w)E\left[\mathbb{I}\left(Z_{w,d}=z\right)\mathbb{I}\left(Z_{w,d}\neq\theta_{B}\right)\right]}{p(w|z)}-\beta=0 $$ $$ p(w|z)\propto \sum_d n(d,w)E\left[\mathbb{I}\left(Z_{w,d}=z\right)\mathbb{I}\left(Z_{w,d}\neq\theta_{B}\right)\right] $$ The expectation above ends up being $$ E\left[\mathbb{I}\left(Z_{w,d}=z\right)\mathbb{I}\left(Z_{w,d}\neq\theta_{B}\right)\right]=E\left[\mathbb{I}\left(Z_{w,d}=z\right)\right] E\left[\mathbb{I}\left(Z_{w,d}\neq\theta_{B}\right)\right]=P\left(Z_{w,d}=z\right |w,d)P\left(Z_{w,d}\neq\theta_{B}|w,d\right)=P\left(Z_{w,d}=z\right|w,d) (1-P\left(Z_{w,d}=\theta_{B}|w,d\right)) $$ |
|||||||||||
|
|
Firstly, thanks for referring to my note. I think the first answer is correct. I'll try to improve the note. |
|||
|
|
