3
$\begingroup$

TL;DR

In the HSLDA paper by Perotte et al, the posterior conditional distribution of $z_{d,n}$ for Gibbs Sampling is specified as:

\begin{equation*} p(z_{d,n}=k| \mathbf{z}_{-d,n}, \mathbf{a}, \mathbf{w}, \eta, \alpha, \beta)\propto \underbrace{\frac{n_{-i, (k)}^{w_i}+\beta}{n_{-i, (k)} + V\beta} \cdot \frac{n_{-i, (d)}^k + \alpha}{n_{-i, (d)} + K\alpha}}_{=p(z_{d,n} | \mathbf{z}_{-d,n}, \mathbf{w}, \alpha, \beta)\text{ (not the problem)}} \cdot \underbrace{\prod_{l \in \mathcal{L}_d} \exp \left\{ -\frac{1}{2}(\bar{\mathbf{z}}_d^T\eta_l - a_{l,d})^2 \right\}}_{ = p(\mathbf{a} | \mathbf{z}, \eta)} \end{equation*}

Isn't the final term constant w.r.t. the word-assignment $z_{d,n} = k$, so we should actually be able to ignore it, right?

Background info on HSLDA:

I'm working through Hierarchically Supervised Latent Dirichlet Allocation by Perotte et al (2011), which is an extension of Blei's LDA. I am having issues understanding the update of the posterior of a the conditional distribution for the Gibbs Sampling procedure. I would like to avoid summarising the entire paper and I'll try to limit this question to only the relevant parts of the paper. Further below, you will find the graphical model and the generative model.

Perotte et al start describing that HSLDA extends LDA in that it allows for $K$ latent topics and document labels. The incorporation of hierarchical document labels is what sets HSLDA apart from other flavours of LDA. The two parts of the models are connected via the word-topic assignments $z_{d,n}$.

HSLDA Graphical Model

Generative Model: HSLDA

The practical use I see in HSLDA, is that it allows to find distinguishing features between document labels that are inherently close to each other, because they are in the same branch in the document label tree.

Deriving the conditional posterior of $z_{d,n}$

is the conditional distribution of the topic-assignment $z_{d,n}$ required to draw samples from the HSLDA posterior distribution using Gibbs sampling and Markov chain Monte Carlo. I have experienced a couple of times now that computer science papers are extremely brief and usually simply state results, rather than derivations. Equation1 in the paper can be derived as follows:

\begin{align*} p(z_{d,n}=k | \mathbf{z}_{-d,n}, \mathbf{a}, \mathbf{w}, \eta, \alpha, \beta) &= \frac{p(z_{d,n}, \mathbf{z}_{-d,n}, \mathbf{a}, \mathbf{w}, \eta, \alpha, \beta)}{p(\mathbf{z}_{-d,n}, \mathbf{a}, \mathbf{w}, \eta, \alpha, \beta)}\\ &=\frac{p(\mathbf{a}, z_{d,n} | \mathbf{z}_{-d,n}, \mathbf{w}, \eta, \alpha, \beta)\cdot p(\mathbf{z}_{-d,n}, \mathbf{w}, \eta, \alpha, \beta)}{p(\mathbf{z}_{-d,n}, \mathbf{a}, \mathbf{w}, \eta, \alpha, \beta)}\\ &\propto p(\mathbf{a}, z_{d,n} | \mathbf{z}_{-d,n}, \mathbf{w}, \eta, \alpha, \beta) \\ &= p(\mathbf{a} | z_{d,n}, \mathbf{z}_{-d,n}, \mathbf{w}, \eta, \alpha, \beta)\cdot p(z_{d,n} | \mathbf{z}_{-d,n}, \mathbf{w}, \eta, \alpha, \beta) \\ &=p(\mathbf{a} | \mathbf{z}, \eta) \cdot p(z_{d,n} | \mathbf{z}_{-d,n}, \mathbf{w}, \alpha, \beta) \end{align*}

By the generative model specification, this results in:

\begin{equation*} p(z_{d,n}=k| \mathbf{z}_{-d,n}, \mathbf{a}, \mathbf{w}, \eta, \alpha, \beta)\propto \underbrace{\frac{n_{-i, (k)}^{w_i}+\beta}{n_{-i, (k)} + V\beta} \cdot \frac{n_{-i, (d)}^k + \alpha}{n_{-i, (d)} + K\alpha}}_{=p(z_{d,n} | \mathbf{z}_{-d,n}, \mathbf{w}, \alpha, \beta)\text{ (not the problem)}} \cdot \underbrace{\prod_{l \in \mathcal{L}_d} \exp \left\{ -\frac{1}{2}(\bar{\mathbf{z}}_d^T\eta_l - a_{l,d})^2 \right\}}_{ = p(\mathbf{a} | \mathbf{z}, \eta)} \end{equation*}

My question

The result is the posterior conditional distribution that we use to sample the values for $z_{d,n} \in \{1,2,...,K\}$ from. Perotte et al keep the last term $p(\mathbf{a} | \mathbf{z}, \eta)$ in the update. However, that term only affects the posterior conditional distribution by a multiplicative constant, because $\bar{\mathbf{z}}_d^T \eta_l$ does not change with $k$ and neither does $a_{l,d}$. Since the sum $\sum_{k=1}^K p(z_{d,n} =k | ...)$ is normalised, it actually does not have any affect at all.

Somehow I feel there should be a term in that posterior conditional distribution, that involves the "bottom part" of the graphical model, but as I described above, this specific term doesn't make sense to me, really. What is the reason that Perotte left that term in the posterior update?

$\endgroup$
1
$\begingroup$

I am not totally sure whether this is a valid answer for your problem, but by definition of the distribution kernel ("...the kernel of a probability mass function is the form of in which any factors that are not functions of any of the variables in the domain are omitted., see Wikipedia") and the definition of the domain of a function (see Wikipedia) we have $z_{d,n}$ being the domain. Now the paper states that $\mathbf{\bar{z}}_d=[\bar{z}_1,...,\bar{z}_K]$ with $\bar{z}_k=N_d^{-1}\sum_{n=1}^{N_d}I(z_{n,d}=k)$ which is clearly a function of the $z_{n,d}$.

Put it otherwise, in order to calculate the normalization constant we must indeed sum/integrate over all possibilities in the domain. If we leave the part $p(\mathbf{a}|\mathbf{z},\eta)$ (that depends on the domain) outside no valid normalization constant will be computed.

As a small example: Let $g(\cdot)$ be some function of $z_{d,n}$ and $\mathcal{S}$ is the space where all $z_{d,n}$ are living. Then it must hold that $p(z_{d,n}|\cdot)\propto g(z_{d,n})$ is equivalent to $p(z_{d,n}|\cdot)=c^{-1} g(z_{d,n})$ with $c=\sum_{z_{d,n} \in\mathcal{S}}g(z_{d,n})$ as we otherwise cannot assure that $\sum_{c \in\mathcal{S}}p(z|\cdot)=1$ holds.

If we know take a partition (as you did): $g(z_{d,n})=g_1(z_{d,n})g_2(z_{d,n})$ and postulate $p(z_{d,n}|\cdot)\propto g_1(z_{d,n})$ we make a mistake whenever $\sum_{z_{d,n} \in\mathcal{S}}g(z_{d,n})\neq\ g_2(z_{d,n})\sum_{c \in\mathcal{S}}g_1(z_{d,n})$.

$\endgroup$
  • $\begingroup$ +1 for your perspective. I have had similar train of thoughts, but never managed to bring it to an end. I think my mistake stems from writing $p(\mathbf{a}|\mathbf{z}, \eta)$ instead of explicitly $p(\mathbf{a}|\mathbf{z}_{-d,n}, z_{d,n}=k)$. When I write it like this, it becomes apparent that I assign $z_{d,n}$ to take different values in $\{1,2,..,K \}$ and $\bar{\mathbf{z}}_d$ will change slightly for every possible topic assignment of $z_{d,n}$. Therefore the problematic term will be different for every $z_{d,n}=k$. $\endgroup$ – KenHBS Nov 14 '17 at 16:32
  • $\begingroup$ Yes, this was definitely the missing piece of the puzzle! $\endgroup$ – KenHBS Nov 14 '17 at 16:38
  • $\begingroup$ I am happy to help. $\endgroup$ – Michael L. Nov 14 '17 at 16:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.