Dirichlet conjugate update derivation

Question

I am attempting to derive the update equations for the conjugate to the Dirichlet distribution, as outlined here: https://mathoverflow.net/questions/20399/conjugate-prior-of-the-dirichlet-distribution

However, parameter update equation I calculate does not match the one suggested there.

My derivation is shown below: \begin{align} f({\theta}|{\alpha}) &= Dir({\theta}|{\alpha})\\ &=\frac{1}{B({\alpha})}\exp(\phi({\alpha})^{T}u({\theta})) \end{align} where, \begin{align} \phi({\alpha})^{T} &= [\alpha_1-1,\cdots,\alpha_K-1]\\ u({\theta}) &= [\ln(\theta_1),\cdots,\ln(\theta_K)]^{T}\\ B({\alpha}) &= \frac{\prod_{i=1}^{K}\Gamma(\alpha_i)}{\Gamma\left(\sum_{i=1}^{K}\alpha_i\right)} \end{align}

Thus, \begin{align} f({\theta}|{\alpha}) &= \frac{1}{B({\alpha})}\exp\left(\sum_{i=1}^{K}\alpha_{i}\ln(\theta_i)-\ln(\theta_i)\right) \end{align}

The exponential family conjugate has form, \begin{align} p({\alpha}|{\nu},\eta) &\propto \frac{1}{B({\alpha})^{\eta}}\exp(\phi({\alpha})^{T}{\nu})\\ &= \frac{1}{B({\alpha})^{\eta}}\exp\left(\sum_{i=1}^{K}\alpha_{i}\nu_{i}-\sum_{i=1}^{K}\nu_i\right)\\ &\propto \frac{1}{B({\alpha})^{\eta}}\exp\left(\sum_{i=1}^{K}\alpha_{i}\nu_{i}\right) \end{align}

Now the posterior update on ${\alpha}$ given ${\theta}$, \begin{align} p({\alpha}|{\theta},{\nu},\eta) &\propto p({\alpha},{\theta}|{\nu},\eta)\\ &= f({\theta}|{\alpha})p({\alpha}|{\nu},\eta)\\ &\propto \left[\frac{1}{B({\alpha})}\exp\left(\sum_{i=1}^{K}\alpha_{i}\ln(\theta_i)-\ln(\theta_i)\right)\right]\times\nonumber\\ &\phantom{{}\propto} \left[\frac{1}{B({\alpha})^{\eta}}\exp\left(\sum_{i=1}^{K}\alpha_{i}\nu_{i}\right) \right]\\ &= \frac{1}{B({\alpha})^{\eta+1}}\exp\left(\sum_{i=1}^{K}\alpha_{i}\ln(\theta_i) + \alpha_{i}\nu_{i}-\ln(\theta_i)\right)\\ \end{align}

Therefore, I get the ${\eta^{t+1}} = {\eta^t} + 1$ update. However, the update on $\nu$ does not match. If we could drop the $-\ln(\theta_i)$, we would get update ${\nu_i^{t+1}} = {\nu_i^t} + \ln(\theta_i)$, which does not match the suggested ${\nu_i^{t+1}} = {\nu_i^t} - \ln(\theta_i)$.

And a follow-up: is there an intuitive meaning behind ${\eta}$ and $\nu$ in this conjugate? ${\eta}$ seems to indicate the level of confidence in the prior, and $\nu$ imposes asymmetry, but some more discussion on this would be appreciated.

Answer 1

There is nothing wrong with this derivation \begin{align} p({\alpha}|{\theta},{\nu},\eta) &\propto p({\alpha},{\theta}|{\nu},\eta)\\ &= f({\theta}|{\alpha})p({\alpha}|{\nu},\eta)\\ &\propto \left[\frac{1}{B({\alpha})}\exp\left(\sum_{i=1}^{K}\alpha_{i}\ln(\theta_i)-\ln(\theta_i)\right)\right]\times\nonumber\\ &\phantom{{}\propto} \left[\frac{1}{B({\alpha})^{\eta}}\exp\left(\sum_{i=1}^{K}\alpha_{i}\nu_{i}\right) \right]\\ &= \frac{1}{B({\alpha})^{\eta+1}}\exp\left(\sum_{i=1}^{K}\alpha_{i}\ln(\theta_i) + \alpha_{i}\nu_{i}-\ln(\theta_i)\right)\\ \end{align} but the part $$\exp\left(-\sum_{i=1}^{K}\ln(\theta_i)\right)$$ does not matter, since it is a multiplicative constant (in $\alpha$) term. Therefore \begin{align} p({\alpha}|{\theta},{\nu},\eta) &\propto \frac{1}{B({\alpha})^{\eta+1}}\exp\left(\sum_{i=1}^{K}\alpha_{i}\{\ln(\theta_i) +\nu_{i}\}\right) \end{align} In conclusion, $$\eta^\text{post}=\eta^\text{prior}+1 \qquad \nu_{i}^\text{post}=\nu_{i}^\text{prior}+\ln(\theta_i) $$ is the correct update. The quoted post has a typo, obviously.

For the follow-up question, I do not think the distribution has an intuitive interpretation.

Answer 2

First of all, exponential family updates are confusing in anything but the natural parametrization where the update rule is just addition. Stick to that parametrization.

I would derive the conjugate prior in this way. The basic idea is the natural parameter values of your conjugate prior distribution $F'$ are $\eta'$ (sums of sufficient statistics of your original distribution $F$). Each observation adds to this vector.

The sufficient statistics for the Dirichlet are $\log x_i$. Therefore, your update rule for its conjugate prior is to sum these up along with an extra parameter that keeps track of how many observations you've summed.

Intuitively, this count parameter is always a concentration parameter; the other parameters are most sensitive to small values of $x_i$. It makes sense that if your Dirichlet sample has a small value of $x_i$, then it probably doesn't have a large $\alpha_i$ compared to the other $\alpha_i$. The converse is maybe less true?

Answer 3

I'm not sure if it makes any intuitive sense, but $\eta$ can be interpreted in a physical context as how far the Dirichlet distribution is from equilibrium, in a micro-canonical sense. See http://arxiv.org/pdf/cond-mat/0603120v1.pdf. This probably isn't the balls-and-urns sort of interpretation that most would find satisfying, but it is an interesting way to look at it!