1
$\begingroup$

I'm trying to work through Coursera's probabilistic graphical models class (week 7: Baeysian prediction) and a have several questions.

  1. In the Dirichlet distribution, I'm having difficulty trying to understand why there's a   -1   in theta's exponent: $$P(\theta)=Dir(\alpha_1, ..., \alpha_k) = \frac{1}{Z} \cdot \prod_{j} \theta_{j}^{\alpha_{j}-1}$$
  2. How do you get from here: $$P(X)=\int_{\theta}P(X|\theta)P(\theta)d\theta$$ to here: $$P(X=x^{i}|\theta) = \int_{\theta} \frac{1}{Z} \cdot \theta_{i} \prod_{j} \theta_{j}^{\alpha_{j}-1}$$
  3. Also, how do you step through the integration for the following?: $$\int_{\theta} \frac{1}{Z} \cdot \theta_{i} \prod_{j} \theta_{j}^{\alpha_{j}-1} = { \alpha_{i}\over{\sum_{j} \alpha_{j}} }$$

These are the lecture notes. My questions refer to the first slide.

$\endgroup$
3
  • $\begingroup$ 1) The $-1$ comes from the definition of Dirichlet distribution. 2) and 3) The notation is horrible (in the slides), there are typos and missinterpretations. Try to find a better reference such as a textbook. $\endgroup$
    – user10525
    Aug 15, 2012 at 18:49
  • $\begingroup$ Is there a good textbook you'd recommend for PGM? $\endgroup$
    – maogenc
    Aug 16, 2012 at 4:15
  • $\begingroup$ did you ask on the course forum ? $\endgroup$ Nov 13, 2012 at 22:03

2 Answers 2

5
$\begingroup$

Just thought I'd add an example of how to calculate the normalising constant. If you know the beta integral, then its easier to use that for direct integration. With a change of variables in the usual definition you get

$$\int_{L}^{U}(x-L)^{a-1}(U-x)^{b-1}dx=(U-L)^{a+b-1}B(a,b)$$

The change in variables is $t=\frac{x-L}{U-L}$ and you get back to the standard definition of the beta integral. To apply this to the calculation of Z we must first determine the limits of integration. This is simple for the simplex as the parameters must all be positive and sum to 1. So we have $$0\leq\theta_1\leq 1$$ $$0\leq\theta_i\leq 1-\sum_{j=1}^{i-1}\theta_j\;\;\; i=2,\dots,n-1$$ $$\theta_n=1-\sum_{j=1}^{n-1}\theta_j $$

This assumes that we integrate in the order $\theta_n,\theta_{n-1},\dots,\theta_1$. The order of integration doesn't matter, but this order is easier to write down.The first integral is a substitution so we have for the second integral.

$$\int_{0}^{1-\sum_{j=1}^{n-2}\theta_j}\left[\prod_{k=1}^{n-2}\theta_{k}^{\alpha_k-1}\right]\theta_{n-1}^{\alpha_{n-1}-1}\left( 1-\sum_{j=1}^{n-2}\theta_j - \theta_{n-1}\right)^{\alpha_n-1}d\theta_{n-1}$$

This is of the form of the transform beta integral with $L=0$ and $U= 1-\sum_{j=1}^{n-2}\theta_j $ hence we get: $$\left[\prod_{k=1}^{n-2}\theta_{k}^{\alpha_k-1}\right]B(\alpha_n,\alpha_{n-1})\left( 1-\sum_{j=1}^{n-2}\theta_j \right)^{\alpha_n+\alpha_{n-1}-1}$$

Now we apply this again to the integral over $\theta_{n-2}$. It is another transformed beta integral but with $U= 1-\sum_{j=1}^{n-3}\theta_j$. Hence we get

$$\left[\prod_{k=1}^{n-3}\theta_{k}^{\alpha_k-1}\right]B(\alpha_n,\alpha_{n-1}) B(\alpha_n+\alpha_{n-1},\alpha_{n-2}) \left( 1-\sum_{j=1}^{n-3}\theta_j \right)^{\alpha_n+\alpha_{n-1}+\alpha_{n-2}-1}$$

It is now straight forward to repeatedly apply this and you get

$$Z= B(\alpha_n,\alpha_{n-1}) B(\alpha_n+\alpha_{n-1},\alpha_{n-2}) B(\alpha_n +\alpha_{n-1}+\alpha_{n-2} ,\alpha_{n-3}) \dots B(\alpha_n+\dots+\alpha_{2},\alpha_1)$$

If you plug in the relation between the beta and gamma integrals $B(a,b)=\frac{\Gamma(a)\Gamma(b)}{\Gamma(a+b)}$ you get the correct normalising constant.

$\endgroup$
3
$\begingroup$

In (2), it should be $P(X = x_i)$, not $P(X = x_i | \theta)$; the equation is arrived at by iterated expectation, i.e. $P(X = x_i) = E[P(X = x_i | \theta)] = E\theta_i = \int \frac 1 Z \theta_i \left(\prod \theta_j ^{\alpha_j - 1}\right) \ d\vec\theta$. (3) is easy if you take the form of the normalizing constant as known; it is $$ \frac 1 {Z(\alpha_1, ..., \alpha_n)} = \frac{\Gamma(\sum \alpha_j)}{\prod \Gamma(\alpha_j)}. $$ The form of the normalizing constant can be found (e.g.) by exploiting the fact that the Dirichlet arises from norming a collection of gamma random variables with shape parameters given by the $\alpha_i$ by their sum. So the integral is $$ \frac{1}{Z(\alpha_1, ..., \alpha_n)} \int \theta_i ^{\alpha_i + 1 - 1} \prod_{i \ne j} \theta_j ^{\alpha_j - 1} \ d\vec\theta = \frac{Z(\alpha_1, ..., \alpha_i + 1, ..., \alpha_n)}{Z(\alpha_1, ..., \alpha_n)} $$ since the integrand is the kernel of a Dirichlet with parameters $(\alpha_1, ..., \alpha_i + 1, ..., \alpha_n)$. Plugging in the form of $Z(\cdot)$ and using the properties of the gamma distribution leads to the result in (3).

In all the integrals it is understood that $\theta_n := 1 - \sum_1 ^ {n - 1} \theta_i$, and the integrals are over the $n - 1$ dimensional simplex.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.