# How do you estimate $\alpha$ parameter of a latent dirichlet allocation model?

Blei has shown that it is possible to estimate $\alpha$ in a LDA model, but I have yet to find a library (any library; C, C++, Java, ...) to do so. Usually, implementations (including Blei's) treat $\alpha$ as a hyperparameter with uniform values, and it is the $\theta$ that is usually estimated, where $\theta$ is a $D \times K$ matrix. It represents each document's topic distribution. However, I am more interested in modeling with the original LDA model where $\alpha$ is used as the parameter for dirichlet distribution of topic distributions, but I am currently stuck at the abyss of mathematical equations in Blei's paper. How should I go on and solve this problem?


[Blei 2003] Blei, David M., Andrew Y. Ng, and Michael I. Jordan. Latent dirichlet allocation. the Journal of machine Learning research 3 (2003): 993-1022.

For fixed vocabulary length $V$ and $K$ topics, $M$ documents, each with $N$ words,

\begin{align} \alpha &\in \R^K_+ & \text{Concentration Parameter} \\ \forall\, k \quad \beta_k &\in \R^{V} & \text{Topic Probability Vectors} \\ \forall\, d \quad \theta_d | \alpha &\sim \Dir(\alpha) & \text{Per-doc Topic Mixing Proportions} \\ \forall\, d,w \quad z_{dw} | \theta_d &\sim \Mult(\theta) & \text{Per-word Topic Indicators} \\ \forall\, d,w \quad w_{dw} | \theta_d &\sim \beta_{z_{dw}} & \text{Words (Observed)} \end{align}

Alternatively, $\beta=(\beta_1,\dots,\beta_K)$ and $\Theta=(\theta_1,\dots,\theta_M)$ can be represented as $K\times V$ and $D\times K$ probability matrices, respectively, where rows sum to one. $\mathbf{Z}=(z_{dj})_{d,j}$ and $\mathbf{W}=(w_{dj})_{d,j}$ can also be represented as matrices.

We model the documents themselves as mixtures of multinomial distributions. The mixing proportions $\theta_d$ for each document $d$ are conditionally independent of the mixing proportions for all other documents, given the concentration parameters $\alpha$. From [Blei 2003]:

The basic idea is that documents are represented as random mixtures over latent topics, where each topic is characterized as a distribution over words.

Typically, one observes the words $\mathbf{W}$ and is interested in inferring the values of the unknown variables $\Theta$, $\mathbf{Z}$ and $\beta$, and optionally the hyperparameters $\alpha$. In the paper, the inference procedure is presented as a Variational Expectation-Maximization algorithm, which essentially divides inference into two steps (c.f. Section 5.3), performed iteratively until convergence:

1. (E-Step) Variational inference to (essentially) estimate the local hidden variables $\Theta$ and $\mathbf{Z}$ given estimates of $\alpha$ and $\beta$. This is done by optimizing over auxiliary variational parameters, whose expectation yields estimates of each hidden variable.
2. (M-Step) Maximum likelihood estimation of global hidden variables $\alpha$ and $\beta$, given estimates of $\Theta$ and $\mathbf{Z}$.

The key here is that estimation on $\alpha$ is done as though we have complete information about the states of the local hidden variables. The hyperparameter $\alpha$ is the concentration parameter to a Dirichlet distribution, from which the multinomial probability vectors $\theta_d$ are "generated".

Crucially, the $\theta_d$ are all conditionally independent given $\alpha$, and $\alpha$ is conditionally independent of all other variables in the model given $\Theta$ (check the plate diagram!). Thus, estimation on $\alpha$ reduces to the problem of estimating the parameter of a Dirichlet distribution given a set of multinomial observations. The following papers derive algorithms to this end:

[Minka 2000] Minka, Thomas. Estimating a Dirichlet distribution. (2000): 3.

[Huang] Huang, Jonathan. Maximum likelihood estimation of Dirichlet distribution parameters.

For an example implementation of the $\alpha$-estimation procedure, you should check out gensim, which directly implements the Newton-Raphson update described in [Huang].

I would definitely recommend reading up about Expectation Maximization and Variational Inference if you aren't already familiar; intuition about these techniques is key to understanding what's happening in Blei's paper.

• After I asked that question, I actually did use dirichlet parameter estimation from per-document topic distribution to get what I wanted, but I needed someone to confirm that it is correct. Thanks for the wrap up. – Kang Min Yoo Dec 26 '14 at 2:18