# Mistake in derivation about categorical distribution and Dirichlet distribution?

$p(x|\pi)$ follows the categorical distribution (the multinomial with one observation), where $\sum\pi_i=1$ and $x$ is a one-hot vector, and $p(\pi|\alpha)$ follows the Dirichlet distribution.

$p(x|\pi)=\prod\pi_i^{x_i}$

$p(\pi|\alpha)=\frac{1}{B(\alpha)}\prod\pi_i^{\alpha_i-1}$

$p(x,\pi|\alpha)=\frac{1}{B(\alpha)}\prod\pi_i^{\alpha_i-1+x_i}$

$p(x|\alpha)=\int p(x,\pi|\alpha)d\pi=\frac{1}{B(\alpha)}\int\prod\pi_i^{\alpha_i-1+x_i}d\pi=\frac{B(\alpha+x)}{B(\alpha)}$

$\sum p(x^{(i)}|\alpha)=\sum\frac{B(\alpha+x^{(i)})}{B(\alpha)}=\sum\frac{\alpha_i+1}{1+\sum\alpha_j}\neq1$

where $x^{(i)}$ is the one-hot vector with the i-th bit equal to 1.

• How do you obtain the second line (the expression for $p(x|\pi)$)? Have you verified that it is a properly normalized probability distribution?
– whuber
May 18, 2016 at 15:19
• @whuber That is the pdf of a categorical distribution (a multinomial with number of trials = 1) May 18, 2016 at 15:29
• @Greenparker Where is it stated that the number of trials is $1$?
– whuber
May 18, 2016 at 15:31
• @whuber hi, sorry I didn't explain it clearly, i think it is the "categorical distribution" or the multinomial with one observation, where $\sum\pi=1$ and $x$ is a one-hot vector. May 18, 2016 at 15:31
• @whuber fair point. I guess I have always run into only one meaning of "Categorical distribution". May 18, 2016 at 15:49

I think the mistake was with simplifying the $B(\cdot)$ function. Suppose $x$ is a $K$ length vecto, and let $\sum_{k=1}^{K} \alpha_k = A$.

$$B(\alpha) = \dfrac{\prod_{k=1}^{K} \Gamma(\alpha_k)}{\Gamma(A )}$$

\begin{align*} \dfrac{B(\alpha+x)}{B(\alpha)} & = \dfrac{\prod_{k=1}^{K} \Gamma(\alpha_k + x_k)}{\Gamma(\sum_{k=1}^{K} \alpha_k + \sum_{k=1}^{K} x_k )}\dfrac{\Gamma(A )} {\prod_{k=1}^{K} \Gamma(\alpha_k)}\\ \end{align*}

Since only one of the components of $x$ can be 1, $\sum_{k=1}^{K} x_k = 1$. Also, note that $\Gamma(z + 1) = z\Gamma(z)$. Using this,

\begin{align*} \dfrac{B(\alpha+x)}{B(\alpha)} & = \dfrac{\prod_{k=1}^{K} \Gamma(\alpha_k + x_k)}{\Gamma(A + 1 )}\dfrac{\Gamma(A )} {\prod_{k=1}^{K} \Gamma(\alpha_k)}\\ & = \dfrac{\prod_{k=1}^{K} \Gamma(\alpha_k + x_k)}{\prod_{k=1}^{K} \Gamma(\alpha_k)}\dfrac{1}{A}\\ & = \dfrac{1}{A} \prod_{k=1}^{K}\dfrac{\Gamma(\alpha_k + x_k)}{\Gamma(\alpha_k)}\\ & = \prod_{k=1}^{K} \dfrac{1}{A} \dfrac{\Gamma(\alpha_k + x_k)}{\Gamma(\alpha_k)} = \prod_{k=1}^{K} p(x_k|\alpha). \end{align*}

To verify if this is a valid pdf \begin{align*} \sum_{k=1}^{K} p(x_k = 1|\alpha_k) & = \sum_{k=1}^{K}\dfrac{1}{A} \dfrac{\Gamma(\alpha_k + 1)}{\Gamma(\alpha_k)}\\ & = \sum_{k=1}^{K}\dfrac{\alpha_k}{A} =1. \end{align*}

I am not sure what steps you skipped, but I think you misused the $\Gamma(\cdot)$ function property.

• thank you so much! I mistakenly assumed that $\Gamma(z+1)=(z+1)\Gamma(z)$. May 18, 2016 at 15:56
• Ah, classic mistake. Glad I could help. May 18, 2016 at 15:59
• so we have $p(x|\alpha)=p(x|E[\pi])$ right? is that some common property? it would be great if you could take a look at my other question, stats.stackexchange.com/q/213389/95569, ty. May 19, 2016 at 2:16