Question about the expression for the generative model in latent Dirichlet allocation (LDA)

Question

I think my question is rather straightforward though I had to include rather a lot of detail for it to make sense.

I'm following Bayesian Methods for Machine Learning course on Coursera and the following expression is given for LDA:

$$ P(W,Z,\theta) = \prod_{d=1}^D P(\theta_d) \prod_{n=1}^{N_d} P(z_{d_{n}} | \theta_d) P(w_{d_{n}} | z_{d_{n}}) $$

where

• D denotes the corpus of documents
• W words in each document
• Z latent variable, topic of each word
• $\theta$ latent variables, distribution of topics per document

In the subsequent video, describing how to derive the variational EM update equations, some of the above terms were substituted and some new notation was introduced:

• $P(\theta_d)$ is substituted by the Dirichlet distribution
• $ \theta_{dz_{dn}} = P(z_{d_{n}} | \theta_d)$
• $ \phi_{z_{dn}w_{dn}} = P(w_{d_{n}} | z_{d_{n}})$

and with this, the previous expression became (ref: start of "LDA: E-step, theta" video)

$$ \log P(\theta, Z, W) = \sum_{n=1}^{N_d} \left[ \sum_{t=1}^{T} (\alpha_t -1) \log \theta_{dt} + \sum_{n=1}^{N_d}\sum_{t=1}^{T} [ z_{dn} = t] \left( \log \theta_{dt} + \log \phi_{{tw}_{{dn}}}\right) \right] + \mathrm{const} $$

Rules of logarithms aside, I'm confused by the second term in the inner sum, in particular, I don't understand how $z_{dn}$ has been dealt with, i.e. how

$$ \prod_{n=1}^{N_d} P(z_{d_{n}} | \theta_d) P(w_{d_{n}} | z_{d_{n}}) $$

is equivalent to $$ \sum_{n=1}^{N_d}\sum_{t=1}^{T} [ z_{dn} = t] \left( \log \theta_{dt} + \log \phi_{{tw}_{{dn}}}\right) $$

References

Bayesian Methods for Machine Learning, Coursera, Week 3, videos: "Latent Dirichlet Allocation" and "LDA: E-step, theta".

Answer 1

All that is going on here is a notational trick - using Iverson brackets to index or "pick out" the appropriate element/row from a vector/matrix of parameters on Multinomial distributions. This arises fairly frequently in the main machine learning textbooks so it's a good one to be aware of. It's also one that got me extremely confused when I first came across it so I hope this will clarify things.

Picking out appropriate elements of a vector/matrix of parameters using Iverson brackets.

The Iverson bracket $[z_{dn} = t]$ is 1 when the latent topic assignment of the $d$th document for the $n$th word $z_{dn}$ is topic $t$, and 0 otherwise.

In order to understand what is going on here, let's explicitly write out the parametric form of the Multinomial* probability distribution on the word-level latent topic assignment indicators $z_{dn}$ using the notation your instructor has chosen:

$$p(z_{dn} | \theta_{d}) = \text{Multinomial}(z_{dn} | \theta_d) = \prod^T_{t=1} (\theta_{dt})^{[z_{dn} = \space t]}$$

In words, this says that the $n$th latent topic assignment of the $d$th document has a Multinomial distribution, parameterised by the vector $\theta_d \in \mathbb{R}^T$. The use of the Iverson bracket (in combination with the product operator) is to "pick out" the appropriate element of this $T$-dimensional parameter vector, so if we request the probability that $z_{dn}$ is topic $t$, we have:

$$\begin{align} p(z_{dn} = t | \theta_d) &= {\theta_{d1}}^{[z_{dn} = \space 1]} \cdot \space ... \space \cdot {\theta_{dt}}^{[z_{dn} = \space t]} \cdot \space ... \space \cdot {\theta_{dT}}^{[z_{dn} = \space T]} \\ &= {\theta_{d1}}^0 \cdot \space ... \cdot \space {\theta_{dt}}^1 \cdot \space ... \cdot \space {\theta_{dT}}^0 \\ &= \theta_{dt} \\ \end{align}$$

Similarly, for the probability of the observed words conditional on the latent topic indicators $w_{dn} | z_{dn}$, we have:

$$p(w_{dn} | z_{dn}, \Phi) = \text{Multinomial}(w_{dn} | z_{dn}, \Phi ) = \prod^T_{t=1} \prod^V_{v=1}(\phi_{tv})^{[z_{dn} = \space t] \cdot [w_{dn} = \space v]} = \prod^T_{t=1} {\phi_{t w_{dn}}}^{[z_{dn} = t]}$$

Here we are using $w_{dn}$ and $z_{dn}$ to pick out the appropriate element of the topic-word distribution matrix $\Phi \in \mathbb{R}^{T \times V}$. Each $(t,v)$-th element $\phi_{tv}$ corresponds to the conditional probability that the $n$th word in the $d$th document is word $v$, given that the topic indicator is topic $t$, that is, $\phi_{tv} = p(w_{dn} = v | z_{dn} = t)$. Together with the product operators, we are using the Iverson bracket $[z_{dn} = t]$ to look up the appropriate row, and $[w_{dn} = v]$ to look up the appropriate column. Instead of using the Iverson bracket $[w_{dn} = v]$ with the product operator $\prod_v$ to look up the appropriate column, it would seem that, similar to the original LDA paper, the instructor treats the word $w_{dn}$ as an (random) index, and writes this as a subscript directly in $\phi_{t w_{dn}}$ of the last equality.

Considering the term of interest and the above parametric functional forms, we have:

$$\begin{align} \log \prod^{N_d}_{n=1} p(z_{dn} | \theta_d) p(w_{dn} | z_{dn}, \Phi) &= \sum^{N_d}_{n=1} \log \left(\prod^T_{t=1} (\theta_{dt})^{[z_{dn} = \space t]} \cdot \prod^T_{t=1} {\phi_{t w_{dn}}}^{[z_{dn} = t]} \right)\\ &= \sum^{N_d}_{n=1} \log \left(\prod^T_{t=1} (\theta_{dt} \phi_{t w_{dn}})^{[z_{dn} = t]} \right)\\ &= \sum^{N_d}_{n=1} \sum^T_{t=1} [z_{dn} = t] (\log \theta_{dt} + \log \phi_{t w_{dn}}) \end{align}$$

As required.