need help understanding Dirichlet (coursera's PGM class week 7 - Bayesian prediction)

Question

I'm trying to work through Coursera's probabilistic graphical models class (week 7: Baeysian prediction) and a have several questions.

1. In the Dirichlet distribution, I'm having difficulty trying to understand why there's a   -1   in theta's exponent: $$P(\theta)=Dir(\alpha_1, ..., \alpha_k) = \frac{1}{Z} \cdot \prod_{j} \theta_{j}^{\alpha_{j}-1}$$
2. How do you get from here: $$P(X)=\int_{\theta}P(X|\theta)P(\theta)d\theta$$ to here: $$P(X=x^{i}|\theta) = \int_{\theta} \frac{1}{Z} \cdot \theta_{i} \prod_{j} \theta_{j}^{\alpha_{j}-1}$$
3. Also, how do you step through the integration for the following?: $$\int_{\theta} \frac{1}{Z} \cdot \theta_{i} \prod_{j} \theta_{j}^{\alpha_{j}-1} = { \alpha_{i}\over{\sum_{j} \alpha_{j}} }$$

These are the lecture notes. My questions refer to the first slide.

Answer 1

Just thought I'd add an example of how to calculate the normalising constant. If you know the beta integral, then its easier to use that for direct integration. With a change of variables in the usual definition you get

$$\int_{L}^{U}(x-L)^{a-1}(U-x)^{b-1}dx=(U-L)^{a+b-1}B(a,b)$$

The change in variables is $t=\frac{x-L}{U-L}$ and you get back to the standard definition of the beta integral. To apply this to the calculation of Z we must first determine the limits of integration. This is simple for the simplex as the parameters must all be positive and sum to 1. So we have $$0\leq\theta_1\leq 1$$ $$0\leq\theta_i\leq 1-\sum_{j=1}^{i-1}\theta_j\;\;\; i=2,\dots,n-1$$ $$\theta_n=1-\sum_{j=1}^{n-1}\theta_j $$

This assumes that we integrate in the order $\theta_n,\theta_{n-1},\dots,\theta_1$. The order of integration doesn't matter, but this order is easier to write down.The first integral is a substitution so we have for the second integral.

$$\int_{0}^{1-\sum_{j=1}^{n-2}\theta_j}\left[\prod_{k=1}^{n-2}\theta_{k}^{\alpha_k-1}\right]\theta_{n-1}^{\alpha_{n-1}-1}\left( 1-\sum_{j=1}^{n-2}\theta_j - \theta_{n-1}\right)^{\alpha_n-1}d\theta_{n-1}$$

This is of the form of the transform beta integral with $L=0$ and $U= 1-\sum_{j=1}^{n-2}\theta_j $ hence we get: $$\left[\prod_{k=1}^{n-2}\theta_{k}^{\alpha_k-1}\right]B(\alpha_n,\alpha_{n-1})\left( 1-\sum_{j=1}^{n-2}\theta_j \right)^{\alpha_n+\alpha_{n-1}-1}$$

Now we apply this again to the integral over $\theta_{n-2}$. It is another transformed beta integral but with $U= 1-\sum_{j=1}^{n-3}\theta_j$. Hence we get

$$\left[\prod_{k=1}^{n-3}\theta_{k}^{\alpha_k-1}\right]B(\alpha_n,\alpha_{n-1}) B(\alpha_n+\alpha_{n-1},\alpha_{n-2}) \left( 1-\sum_{j=1}^{n-3}\theta_j \right)^{\alpha_n+\alpha_{n-1}+\alpha_{n-2}-1}$$

It is now straight forward to repeatedly apply this and you get

$$Z= B(\alpha_n,\alpha_{n-1}) B(\alpha_n+\alpha_{n-1},\alpha_{n-2}) B(\alpha_n +\alpha_{n-1}+\alpha_{n-2} ,\alpha_{n-3}) \dots B(\alpha_n+\dots+\alpha_{2},\alpha_1)$$

If you plug in the relation between the beta and gamma integrals $B(a,b)=\frac{\Gamma(a)\Gamma(b)}{\Gamma(a+b)}$ you get the correct normalising constant.

Answer 2

In (2), it should be $P(X = x_i)$, not $P(X = x_i | \theta)$; the equation is arrived at by iterated expectation, i.e. $P(X = x_i) = E[P(X = x_i | \theta)] = E\theta_i = \int \frac 1 Z \theta_i \left(\prod \theta_j ^{\alpha_j - 1}\right) \ d\vec\theta$. (3) is easy if you take the form of the normalizing constant as known; it is $$ \frac 1 {Z(\alpha_1, ..., \alpha_n)} = \frac{\Gamma(\sum \alpha_j)}{\prod \Gamma(\alpha_j)}. $$ The form of the normalizing constant can be found (e.g.) by exploiting the fact that the Dirichlet arises from norming a collection of gamma random variables with shape parameters given by the $\alpha_i$ by their sum. So the integral is $$ \frac{1}{Z(\alpha_1, ..., \alpha_n)} \int \theta_i ^{\alpha_i + 1 - 1} \prod_{i \ne j} \theta_j ^{\alpha_j - 1} \ d\vec\theta = \frac{Z(\alpha_1, ..., \alpha_i + 1, ..., \alpha_n)}{Z(\alpha_1, ..., \alpha_n)} $$ since the integrand is the kernel of a Dirichlet with parameters $(\alpha_1, ..., \alpha_i + 1, ..., \alpha_n)$. Plugging in the form of $Z(\cdot)$ and using the properties of the gamma distribution leads to the result in (3).

In all the integrals it is understood that $\theta_n := 1 - \sum_1 ^ {n - 1} \theta_i$, and the integrals are over the $n - 1$ dimensional simplex.