7
$\begingroup$

I am attempting to derive the update equations for the conjugate to the Dirichlet distribution, as outlined here: https://mathoverflow.net/questions/20399/conjugate-prior-of-the-dirichlet-distribution

However, parameter update equation I calculate does not match the one suggested there.

My derivation is shown below: \begin{align} f({\theta}|{\alpha}) &= Dir({\theta}|{\alpha})\\ &=\frac{1}{B({\alpha})}\exp(\phi({\alpha})^{T}u({\theta})) \end{align} where, \begin{align} \phi({\alpha})^{T} &= [\alpha_1-1,\cdots,\alpha_K-1]\\ u({\theta}) &= [\ln(\theta_1),\cdots,\ln(\theta_K)]^{T}\\ B({\alpha}) &= \frac{\prod_{i=1}^{K}\Gamma(\alpha_i)}{\Gamma\left(\sum_{i=1}^{K}\alpha_i\right)} \end{align}

Thus, \begin{align} f({\theta}|{\alpha}) &= \frac{1}{B({\alpha})}\exp\left(\sum_{i=1}^{K}\alpha_{i}\ln(\theta_i)-\ln(\theta_i)\right) \end{align}

The exponential family conjugate has form, \begin{align} p({\alpha}|{\nu},\eta) &\propto \frac{1}{B({\alpha})^{\eta}}\exp(\phi({\alpha})^{T}{\nu})\\ &= \frac{1}{B({\alpha})^{\eta}}\exp\left(\sum_{i=1}^{K}\alpha_{i}\nu_{i}-\sum_{i=1}^{K}\nu_i\right)\\ &\propto \frac{1}{B({\alpha})^{\eta}}\exp\left(\sum_{i=1}^{K}\alpha_{i}\nu_{i}\right) \end{align}

Now the posterior update on ${\alpha}$ given ${\theta}$, \begin{align} p({\alpha}|{\theta},{\nu},\eta) &\propto p({\alpha},{\theta}|{\nu},\eta)\\ &= f({\theta}|{\alpha})p({\alpha}|{\nu},\eta)\\ &\propto \left[\frac{1}{B({\alpha})}\exp\left(\sum_{i=1}^{K}\alpha_{i}\ln(\theta_i)-\ln(\theta_i)\right)\right]\times\nonumber\\ &\phantom{{}\propto} \left[\frac{1}{B({\alpha})^{\eta}}\exp\left(\sum_{i=1}^{K}\alpha_{i}\nu_{i}\right) \right]\\ &= \frac{1}{B({\alpha})^{\eta+1}}\exp\left(\sum_{i=1}^{K}\alpha_{i}\ln(\theta_i) + \alpha_{i}\nu_{i}-\ln(\theta_i)\right)\\ \end{align}

Therefore, I get the ${\eta^{t+1}} = {\eta^t} + 1$ update. However, the update on $\nu$ does not match. If we could drop the $-\ln(\theta_i)$, we would get update ${\nu_i^{t+1}} = {\nu_i^t} + \ln(\theta_i)$, which does not match the suggested ${\nu_i^{t+1}} = {\nu_i^t} - \ln(\theta_i)$.

And a follow-up: is there an intuitive meaning behind ${\eta}$ and $\nu$ in this conjugate? ${\eta}$ seems to indicate the level of confidence in the prior, and $\nu$ imposes asymmetry, but some more discussion on this would be appreciated.

$\endgroup$
  • $\begingroup$ I added an answer, but do you really need the conjugate prior? You don't if you're doing maximum likelihood estimation. $\endgroup$ – Neil G Nov 24 '15 at 20:31
5
$\begingroup$

There is nothing wrong with this derivation \begin{align} p({\alpha}|{\theta},{\nu},\eta) &\propto p({\alpha},{\theta}|{\nu},\eta)\\ &= f({\theta}|{\alpha})p({\alpha}|{\nu},\eta)\\ &\propto \left[\frac{1}{B({\alpha})}\exp\left(\sum_{i=1}^{K}\alpha_{i}\ln(\theta_i)-\ln(\theta_i)\right)\right]\times\nonumber\\ &\phantom{{}\propto} \left[\frac{1}{B({\alpha})^{\eta}}\exp\left(\sum_{i=1}^{K}\alpha_{i}\nu_{i}\right) \right]\\ &= \frac{1}{B({\alpha})^{\eta+1}}\exp\left(\sum_{i=1}^{K}\alpha_{i}\ln(\theta_i) + \alpha_{i}\nu_{i}-\ln(\theta_i)\right)\\ \end{align} but the part $$\exp\left(-\sum_{i=1}^{K}\ln(\theta_i)\right)$$ does not matter, since it is a multiplicative constant (in $\alpha$) term. Therefore \begin{align} p({\alpha}|{\theta},{\nu},\eta) &\propto \frac{1}{B({\alpha})^{\eta+1}}\exp\left(\sum_{i=1}^{K}\alpha_{i}\{\ln(\theta_i) +\nu_{i}\}\right) \end{align} In conclusion, $$\eta^\text{post}=\eta^\text{prior}+1 \qquad \nu_{i}^\text{post}=\nu_{i}^\text{prior}+\ln(\theta_i) $$ is the correct update. The quoted post has a typo, obviously.

For the follow-up question, I do not think the distribution has an intuitive interpretation.

$\endgroup$
  • $\begingroup$ Thanks. That makes sense. However, the post I referenced shows the posterior update as $\nu_{i}^\text{post}=\nu_{i}^\text{prior}-\ln(\theta_i)$, not $\nu_{i}^\text{post}=\nu_{i}^\text{prior}+\ln(\theta_i)$. Are the elements of $\nu$ assumed negative for some reason? $\endgroup$ – sr71 Nov 24 '15 at 20:15
  • $\begingroup$ The original post made a mistake in the sign, that's all... $\endgroup$ – Xi'an Nov 24 '15 at 20:37
4
$\begingroup$

First of all, exponential family updates are confusing in anything but the natural parametrization where the update rule is just addition. Stick to that parametrization.

I would derive the conjugate prior in this way. The basic idea is the natural parameter values of your conjugate prior distribution $F'$ are $\eta'$ (sums of sufficient statistics of your original distribution $F$). Each observation adds to this vector.

The sufficient statistics for the Dirichlet are $\log x_i$. Therefore, your update rule for its conjugate prior is to sum these up along with an extra parameter that keeps track of how many observations you've summed.

Intuitively, this count parameter is always a concentration parameter; the other parameters are most sensitive to small values of $x_i$. It makes sense that if your Dirichlet sample has a small value of $x_i$, then it probably doesn't have a large $\alpha_i$ compared to the other $\alpha_i$. The converse is maybe less true?

$\endgroup$
1
$\begingroup$

I'm not sure if it makes any intuitive sense, but $\eta$ can be interpreted in a physical context as how far the Dirichlet distribution is from equilibrium, in a micro-canonical sense. See http://arxiv.org/pdf/cond-mat/0603120v1.pdf. This probably isn't the balls-and-urns sort of interpretation that most would find satisfying, but it is an interesting way to look at it!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.