Latent Dirichlet Allocation - dimensionality of the Dirichlet prior parameter

Question

I seeking some clarity on the dimensionality of the (hyper)parameter $\eta$ of the "smoothed LDA" model in Section 5.4 of the original paper by Blei, Ng, Jordan (2003), which can be found here.

Queries.

If each of the $K$ topic-specific word-distribution row vectors $\beta_{k, \space :} \in [0, 1]^V$ are "independently drawn from an exchangeable Dirichlet distribution" [with parameter $\eta$], then are each of the $\beta_{k, \space :}$ parametrised by an identical scalar parameter $\eta$, or is it the case that each $\beta_{k, \space :}$ has its own scalar parameter $\eta_k$?

What is the dimensionality of $\eta$ in the plate diagram below?

Context.

Here is the plate diagram of the smoothed LDA model:

The relevant section from the paper:

In the LDA setting, we obtain the extended graphical model shown in Figure 7 [above]. We treat $\boldsymbol{\beta}$ as a $k \times V$ random matrix (one row for each mixture component), where we assume that each row is independently drawn from an exchangeable Dirichlet distribution. We now extend our inference procedures to treat the $\beta_i$ as random variables that are endowed with a posterior distribution, conditioned on the data.

An explanatory footnote goes on to say:

An exchangeable Dirichlet is simply a Dirichlet distribution with a single scalar parameter $\eta$. The density is the same as a Dirichlet (Eq. 1) where $\alpha_i = \eta$ for each component.

Now (Eq. 1) referenced in the explanatory footnote is similar in that we impose a Dirichlet prior with parameter vector $\alpha = (\alpha_1, ..., \alpha_K)$ on the document-specific topic-distribution vector $\mathbf{\theta} \in (0, 1)^K$:

$$p(\theta | \alpha) = \frac{\Gamma(\sum^K_{i=1} \alpha_i)}{\prod^k_{i=1} \Gamma(\alpha_i)} {\theta_1}^{\alpha_1 - 1} \cdot \space ... \space \cdot {\theta_K}^{\alpha_K - 1}$$

Now following the reasoning set out in the footnote (where the authors use $\beta_i$ to represent what I have written as $\beta_{k, :}$) I have that:

$$p(\beta_{k, \space :} | \eta) = \frac{\Gamma(\sum^V_{v=1} \eta)}{\prod^V_{v=1} \Gamma(\eta)} {\beta_{k1}}^{\eta - 1} \cdot \space ... \space \cdot {\beta_{kV}}^{\eta - 1} = \frac{\Gamma(\eta V)}{\Gamma(\eta)^V} \prod^V_{v=1} {\beta_{kv}}^{\eta - 1}$$

Which leads me to the question I've specified above.

I am aware that if we use independent and identically distributed (IID) random variables as a reference point, exchangeable random variables are identically distributed but not independent. If this is the sense in which the authors mean that the vector $\beta_{k, \space :}$ is "independently drawn from an exchangeable Dirichlet distribution", then my guess would be that it would be that each $\beta_{k, \space :}$ is identically parametrised by a single scalar $\eta$. However I am not sure, partly because the definition I have listed is not sufficiently precise.

Clarity would be appreciated.

Answer 1

The question you raised is down to confusion concerning whether or not the $\beta_{k, \space :}$ is drawn from an identically parametrised Dirichlet distribution, or whether each $\beta_{k, \space :}$ is drawn from a uniquely parametrised Dirichlet distribution. There is no reason to assume the latter, and the concern is entirely separate to the question of whether a Dirichlet distribution is parametrised by a scalar or a vector - the authors mean that $\eta$ is a scalar (hyper)parameter. And hence it is a scalar in the plate diagram.

To emphasise the distinction between an exchangeable Dirichlet prior, parametrised by a scalar $\eta$, and a Dirichlet prior that is not exchangeable, that is parametrised by a vector $\boldsymbol{\eta} \in \mathbb{R}^V$, consider how the latter can "collapse" into the former.

The Dirichlet prior is:

$p(\beta_{k, \space :} | \boldsymbol{\eta}) = \frac{\Gamma(\sum^V_{v=1} \eta_v)}{\prod^V_{v=1} \Gamma(\eta_v)} {\beta_{k, 1}}^{\eta_1 - 1} \cdot \space ... \space \cdot {\beta_{k, V}}^{\eta_V - 1}$

Now setting all elements of $\eta_v$ of $\boldsymbol{\eta}$ to be the same scalar parameter $\eta$, we have the exchangeable Dirichlet prior:

$p(\beta_{k, \space :} | \eta) = \frac{\Gamma(\eta V)}{\Gamma(\eta)^V}{\beta_{k, 1}}^{\eta - 1} \cdot \space ... \space \cdot {\beta_{k, V}}^{\eta - 1}$

The paper additionally confirms this at the end of Appendix A.4.2, where it makes reference to a finding "an empirical Bayes point estimate of $\eta$, the scalar parameter for the exchangeable Dirichlet in the smoothed LDA model in Section 5.4."