2
$\begingroup$

When reading about the LDA, the generative procedure for a document is often presented as follows:

  1. For each topic $k\in \{1,\ldots, K\}$
    • Draw a distribution over words $\phi_k\sim \text{Dirichlet}(\alpha)$
  2. For each article $m\in\{1,\ldots,M\}$
    • Draw a vector of topic probabilities $\theta_m\sim\text{Dirichlet}(\beta)$
    • For each word entry $n\in\{1,\ldots,N_m\}$
      • Draw a topic assignment $z_{m,n}\sim \text{Multinomial}(\theta_m)$, $z_{m,n}\in\{1,\ldots,K\}$
      • Draw a word $w_{m,n}\sim \text{Multinomial}(\phi_{z_{m,n}})$

Having estimated my own LDA-model on a textual corpus, there is one point I don't quite get here: My estimated topics are distributions over words, but the distributions differ among the topics - some are sharply peaked around only a few words, while others are more broadly distributed over words. This is despite having fixed $\alpha$ to be a scalar.

My question is: When I fix $\alpha$ to be a scalar, then how come the profile of word-distributions of my topics can differ? How can some be sharply peaked, but others more broad? I would think that the profiles should be more or less identical since $\alpha$ is a scalar.

$\endgroup$
8
  • 1
    $\begingroup$ Are you estimating the hyperparameter $\alpha$ using an empirical Bayes like procedure, or are you fixing in advance then running the topic modelling? $\endgroup$
    – microhaus
    Commented Apr 21, 2021 at 16:31
  • 1
    $\begingroup$ @microhaus I am fixing the hyperparameters in advance (setting them to a scalar) $\endgroup$
    – Tyler D
    Commented Apr 22, 2021 at 6:59
  • 1
    $\begingroup$ Are you also fixing the hyperparameter $\beta$ in advance also and setting it to a scalar for the duration of the algorithm? $\endgroup$
    – microhaus
    Commented Apr 23, 2021 at 1:50
  • 1
    $\begingroup$ There is a small issue with your understanding of how the hyperparameter $\boldsymbol{\alpha}$, be it scalar or vector, encodes prior beliefs about topic vectors $\phi_k$. I felt that it might be instructive to address this, so have included a partial answer on this point alone. The crux of your question on the profile of the topic vectors $\phi_k$ generated after approximate posterior inference can only be addressed generally, unless an exhaustive analysis is carried out with data. $\endgroup$
    – microhaus
    Commented Apr 23, 2021 at 1:57
  • 1
    $\begingroup$ Furthermore, it's also difficult to say anything specific about this unless more context is supplied on the kind of variational inference that is being implemented in your software package, or what paper the package implements (it will likely be online VI by Hoffman et al.). For this reason, I would be happy to try and address the core of the question after you have supplied further info on the paper or software package you are using. Even in this case only general comments can be supplied. $\endgroup$
    – microhaus
    Commented Apr 23, 2021 at 2:04

1 Answer 1

1
$\begingroup$

I think it would be helpful in this instance to draw a distinction between:

1. How the hyperparametrisation encodes prior beliefs about the topic vectors $\phi_k$ sampled from the Dirichlet prior $p(\phi_k | \alpha)$, before any inference is conducted.

2. The outcome of approximate posterior inference (either estimated topic vectors or a distribution on topic vectors, depending on further context).

I will focus on 1. as your question suggests that there is something you might have missed in your understanding of this.

I cannot appropriately comment on 2. in any specificity as it would require more context on the kind of variational inference being implemented under the hood of the software package you are using, and because it's difficult to say anything specific without recourse to the data. This is where the crux of your question lies, so I will try and update this answer after more context has been supplied, as requested in comments.

Characterising sampled topic vectors $\phi_k$ from a Dirichlet prior with varying hyperparametrisations $\boldsymbol{\alpha}$.

In response to:

How can some [profiles of topics $\phi_k$] be sharply peaked, but others more broad? I would think that the profiles should be more or less identical since $\alpha$ is a scalar.

What follows is not strictly applicable to your question, because your question is about the profile of the $\phi_k$ after point 2. above has happened. However, in order to interpret the profile of the $\phi_k$ after approximate posterior inference, a clear understanding of how prior beliefs about $\phi_k$ are encoded is important.

Notationally I am going to define $\boldsymbol{\alpha} \in \mathbb{R}^V$ to be a vector hyperparameter, with $V$ positive components $\alpha_v > 0$ for each of $V$ vocabulary words.

When we specify that $\boldsymbol{\alpha}$ is a scalar $\alpha$, we mean that all for all $V$ components, $\alpha_v = \alpha$. When this is the case we refer to it as a symmetric Dirichlet distribution, or in the terminology of the original paper, an exchangeable Dirichlet distribution (see footnote 2 on page 1006).

Furthermore, I am going to refer to the 'profile' of a topic vector $\phi_k \in [0, 1]^V$ in terms of sparsity. So a sparse topic vector would mean that probability mass is sharply concentrated on a few components say $\phi_{kv}$ and $\phi_{ku}$, and the rest are close to 0 (hence sparse). Metaphorically, the landscape of the probability mass function is one of 'sharp peaks'. On the other hand when the topic vector is not sparse, probability mass is evenly distributed uniformly over all components of the vector $\phi_k$, yielding a 'flat plain'.

With these distinctions in mind:

1. For symmetric Dirichlet distributions, it is the magnitude of the scalar $\alpha$ that controls the sparsity of the topic vector $\phi_k$.

What you have omitted is that sparsity is not encoded by whether a Dirichlet distribution is symmetric alone, rather it is encoded by the magnitude of $\alpha$.

In your context, the symmetric Dirichlet distribution would be:

$$p(\phi_k | \alpha) = \frac{\Gamma(\alpha V)}{\Gamma(\alpha)^V} \prod^V_{v=1} \phi_{kv}^{\alpha -1}$$

In the sub-heading "Special cases" in the Dirichlet distribution wikipedia article, the critical value to be aware of is that sparsity refers to situations where $\alpha < 1$.

Here is a plot for $K = 11$ topic vectors $\phi_k$ sampled from a symmetric Dirichlet prior, and where each $\phi_k$ is a distribution over $V = 10$ words in the sparse $\alpha = 0.01$ case:

enter image description here

Note that we can control the sparsity or 'sharp peak profile' of the sampled topic vector $\phi_k$ as a general characteristic only in a probabilistic sense. You can see this by noting that even though many topics $\phi_k$ for $\alpha = 0.01$ have probability mass concentrated on one word, not all possess this characteristic, e.g. $\phi_6$ has two 'peaks. A helpful mental model for this probabilistic aspect is to conceive of drawing a topic vector $\phi_k$ from a bag of topic vectors.

Furthermore, notice that we cannot control the exact location of the peak in the symmetric case, that is, the location where probability mass is concentrated, only the general profile.

Now contrast this with the non-sparse case $\alpha = 10$, the 'flat plain' profile:

enter image description here

2. When the Dirichlet distribution is not symmetric, that is, using a hyperparameter $\boldsymbol{\alpha}$ with non-identical components $\alpha_v$, we can encode prior beliefs over favouring one or more components $\phi_{kv}$ in the probability vector $\phi_{k}$ over other components.

The Dirichlet prior is now:

$$p(\phi_k | \boldsymbol{\alpha}) = \frac{\Gamma(\sum^V_{v=1} \alpha_v)}{\prod^V_{v=1} \Gamma(\alpha_v)} \prod^V_{v=1} \phi_{kv}^{\alpha_v - 1}$$

In this case, we can specify where probability mass is concentrated in each topic vector $\phi_k$. And this is done by adjusting the relative magnitude of each component $\alpha_v$, and where this is relative to other components $\alpha_{-v}$. In this way, we can specify where the the 'peaks' are located in each topic vector $\phi_k$, so if we want to place more probability mass on word $v$, i.e. making $\phi_{kv}$ larger, we would make $\alpha_v$ large relative to $\alpha_{-v}$. The reservation that this correspondence is probabilistic in nature similarly holds.

An example with $\boldsymbol{\alpha}$ with the last component $\alpha_{v} = 1$, and all other components $\alpha_{-v} = 0.01$:

enter image description here

Topic vectors $\phi_k$ after approximate posterior inference.

Here is a partial answer to why it may not be the case that the kinds of topic vectors $\phi_k$ on which you encode prior beliefs on in 1. will be the same after you have conducted approximate posterior inference in 2. This can only be at a high-level, and there are both conceptual and implementational details that need to be glossed over.

i. You fix scalar hyperparameters $\alpha = 0.1$ and $\beta = 0.1$ (I think these are the gensim default settings). The former encodes a prior belief about sparsity in your topic vectors $\phi_k$. The latter encodes a prior belief about sparsity in your document topic mixture proportions $\theta_m$.

ii. Posterior inference means that you want to compute a posterior distribution over latent variables $\Phi = \{ \phi_1, \cdots \phi_K \}$, $\Theta = \{\theta_1, \cdots, \theta_M \}$, and $Z = \{ z_1 \cdots, z_M \}$ given documents $W = \{w_1, \dots, w_M \}$.

$$p(\Phi, \Theta, Z | W ; \alpha, \beta) = \frac{p(\Phi, \Theta, Z, W ; \alpha, \beta)}{p(W ; \alpha, \beta)}$$

To see how your prior beliefs on all $K$ topic vectors $\phi_k$ influences this computation, note that we can use Bayes rule and the way the LDA model factorises to get

$$p(\Phi, \Theta, Z | W ; \alpha, \beta) = \frac{p(Z, W | \Theta, \Phi)}{p(W | \alpha, \beta)} p(\Theta ; \beta) \cdot p(\Phi ; \alpha)$$

Note that $p(\Phi ; \alpha) = \prod^K_{k=1} p(\phi_k ; \alpha)$ and $p(\Theta | \beta) = \prod^M_{m=1} p(\theta_m ; \beta)$ are your priors, $p(Z , W| \Theta, \Phi)$ can be thought of as a likelihood function on the corpus $W$ (observed data) and the corpus topic assignments $Z$ (latent variables), and $p(W ; \alpha, \beta)$ is a normalisation constant.

Ideally, in standard Bayesian statistical settings, given prior beliefs on our parameters $\Phi$ and $\Theta$, we 'confront' these beliefs with the data $W$ using a likelihood function $p(Z, W | \Theta, \Phi)$ to compute a posterior distribution over our parameters $\Phi$ and $\Theta$, and in this case the latent variables $Z$ also.

It is precisely this confrontation with the data $W$ through the likelihood function that implies that in general, we have no reason to expect that our prior beliefs on the topic vectors $\phi_k$ in $\Phi$ will be necessarily be the same as our posterior beliefs about the topic vectors $\phi_k$.

And that is the crux of why encoding prior beliefs about the $\phi_k$ being sparse may not necessarily mean that our posterior beliefs are that the $\phi_k$ are sparse - it depends on, amongst other things, the data $W$.

iii. Strictly, however, the situation is not as simple as I have laid out. The departure from the above idealised setting, which complicates things, but does not change what I have stated in italics, is the inclusion of the latent topic assignments $Z$. This inclusion makes computing the normalisation constant $p(W ; \alpha, \beta)$ intractable, thereby making the posterior $p(\Phi, \Theta, Z | W ; \alpha, \beta)$ difficult to compute.

The gist of what is going on under the hood with gensim is that we instead compute an approximation to this posterior, with the following factorisation structure

\begin{align*}q(\Phi, \Theta, Z ; \Lambda, \Gamma, \Sigma) &= q(\Phi; \Lambda) q(\Theta; \Gamma) q(Z ; \Sigma) \\ &= q(\Theta, \Gamma) q(Z; \Sigma) \prod^K_{k=1} q(\phi_k ; \lambda_k) \end{align*}

Where I have not shown further factorisation on $q(\Theta; \Gamma)$ and $q(Z; \Sigma)$. The above factorisation structure is a feature of the fact that we assume that our approximation lies in the mean field family of distributions $\mathcal{Q}$. The functional forms we assume on each of the $q(\Theta, \Gamma)$ etc. are chosen to match those specified in the LDA model.

Now when you do variational inference, which is what is going under the hood with gensim, you are computing estimates of the variational parameters $\hat{\Lambda} = \{\hat{\lambda}_1, \cdots, \hat{\lambda}_K \}$ (and also estimates $\hat{\Gamma}$ and $\hat{\Sigma}$) so that the posterior approximation $q$ is "close" to the true posterior $p$, in the sense of minimising KL-divergence

$$\hat{\Lambda}, \hat{\Gamma}, \hat{\Sigma} = \underset{\Lambda, \Gamma, \Sigma}{\text{argmin}} \space D_{KL}(q(\Phi, \Theta, Z; \Lambda, \Gamma, \Sigma \space ) \Vert \space p(\Phi, \Theta, Z | W ; \alpha, \beta))$$

Meaning that at the end of training, we get an approximation to the posterior $q(\Phi, \Theta, Z; \hat{\Lambda}, \hat{\Gamma}, \hat{\Sigma})$. Specfically with respect to topic vectors $\phi_k$, we get $K$ invidual posterior distributions over our topic vectors $\phi_k$ in the form of $K$ Dirichlet distributions $q(\phi_1; \hat{\lambda}_1), \dots, q(\phi_K, \hat{\lambda}_K)$.

The difference here from iii. is that we are computing an approximate posterior $q$ through estimation of $\Lambda, \Gamma, \Sigma$ rather than an exact posterior $p$.

The similarity with iii. however is that in the process of estimating $\Lambda, \Gamma, \Sigma$, thereby indexing an approximate posterior $q$ so as to minimise a "distance" (divergence measure) between a family of approximate posteriors $\mathcal{Q}$ and the true posterior $p$; there is still confrontation with data $W$, albeit indirectly through the likelihood $p(Z, W | \Theta, \Phi)$ in the true posterior $p(\Phi, \Theta, Z | W, \alpha, \beta)$ in $D_{KL}(q || p)$.

Hence in the variational Bayesian inference case, it will be via this transmission mechanism involving data $W$ which will be the primary reason for getting posteriors on $\phi_k$ that differ from your prior beliefs.

iv. Some final remarks.

  • The fact that gensim implements online variational inference for LDA in the sense of Hoffman et al. (2010) rather than the batch LDA of Blei et al. (2003) does not change the general answer to your question very much. The minor difference is that stochastic approximation is used to estimate $\Lambda$ using sampling on single documents or on mini-batches of documents, which introduces noise.

  • Strictly, the online variational inference paper states that you compute posteriors on $\phi_k$, parametrised by estimates $\hat{\lambda_k}$ of the variational parameters. At this stage, and without further scrutiny of the code under the hood of gensim, I cannot comment on how exactly gensim is producing topic vectors $\phi_k$ from these posterior distributions $q(\phi_k ; \hat{\lambda}_k)$, but this is minor and does not alter any of what I have written to address your question.

  • This all assumes you are not doing empirical Bayes/type-II maximum likelihood estimation of the prior hyperparameters $\alpha$, $\beta$.

$\endgroup$
1
  • 1
    $\begingroup$ Thanks a lot for your effort @microhaus - this is very valuable and much appreciated. Now I have a much better grasp of what goes on under the hood. $\endgroup$
    – Tyler D
    Commented Apr 26, 2021 at 8:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.