1
$\begingroup$

I'm working on a Gibbs sampler for a (somewhat custom version of) Latent Dirichlet Allocation model.

In short, I have data that comes from a $K$-dimensional Dirichlet-Multinomial distribution, i.e. $$\theta_d \sim \text{Dirichlet}(\boldsymbol{\alpha}), \quad \forall d = 1,\ldots,D$$ $$z_{d,n} \sim \text{Categorical}(\theta_d), \quad \forall n = 1,\ldots,N_d$$ where $z_{d,n}\in[1,K]$, $\theta_d$ is a $K$ dimensional probability vector, and in principle $\boldsymbol{\alpha}=(\alpha_1,\ldots,\alpha_K)$ is a $K$ dimensional vector of positive numbers. Additionally, I have a hyper-prior on $\boldsymbol{\alpha}$: $$\alpha_k \sim \text{Gamma}(a,b), \quad \forall k=1,\ldots,K$$ Oddly enough, I had no problem finding a way to estimate $\boldsymbol{\alpha}$ in such case, i.e. finding maximum of $p(\boldsymbol{\alpha}|\mathcal{Z})$ - it is done through fixed-point iteration, as presented in Wallach (2006), p. 39., where the new value is estimated as follows: $${\alpha_K}^*=\alpha_k \frac{\sum_{d=1}^D \Big( \Psi(N_{k|d} + \alpha_k) - \Psi(\alpha_k) \Big) + a}{\sum_{d=1}^D \Big( \Psi(N_d + \sum_{k=1}^K\alpha_k) - \Psi(\sum_{k=1}^K\alpha_k) \Big) - \frac{1}{b}}$$ where $N_{k|d}$ is the number of $z_{d,n}$'s in $d$ that where assigned to $k$ and $\Psi$ is the digamma function.

What I need, is a method for finding an analogous estimate but under the assumption that all elements of $\boldsymbol{\alpha}$ are equal. This seems like a simpler case and in LDA is often assumed for the topic-term distributions. Nevertheless, for the life of me I cannot find it anywhere, and I am unable to derive it myself - intuitively, averaging $\alpha_k$'s from the above method makes sense, but I am not sure and perhaps there is a more computationally efficient algorithm.

$\endgroup$

1 Answer 1

1
$\begingroup$

You can assume a Gamma prior on $\alpha$ and estimate the hyperparameter using a Metropolis-Hastings step within your Gibbs sampler. The standard reference would be Escobar & West (1995). See also Eq. 10 in this paper.

You need to take the Dirichlet likelihood, combine it with your Gamma prior and evaluate the ratio between the two. It is straight-forward to do so assuming that the elements in $\alpha$ are the same for all $k$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.