Properties of conditional probability distributions

Question

This is a problem from a machine learning pset that I'm self-learning from http://www.seas.harvard.edu/courses/cs281/assignment-1.pdf.

Suppose we are provided with a hierarchy of three distributions: $p(\alpha)$, $p(\theta | \alpha)$ and $p(y | \theta, \alpha)$. Write expressions to compute the following related distributions:

$p(y|\theta)$, $p(y|\alpha)$, $p(y)$, $p(\theta|y,\alpha)$, $p(\alpha|y,\theta)$, $p(\theta)$, $p(\theta|y)$, $p(\alpha|y)$

I see that I integrate over the joint distribution to get the marginal probability

$$p(y|\theta) = \int p(y|\theta,\alpha)p(\alpha)d\alpha$$

This gives me the first expression in terms of the givens.

However, I don't know how to get the others. I know I can set up various equations using Bayes' Theorem

$$p(\theta|\alpha) = p(\alpha|\theta) * p(\theta)/p(\alpha)$$

But the above equation has two unknowns ($p(\theta)$ and $p(\alpha|\theta)$). Is there another property that I can exploit besides marginalizing the joint distribution and utilizing Bayes' Theorem?

Answer 1

The given conditionals allow you to find the joint density of the three random variables using the product rule $$ p(y,\theta,\alpha)=p(y\mid\theta,\alpha) \, p(\theta\mid\alpha) \, p(\alpha) \, . $$

(Of course this is a sloppy notation, since the $p$'s are not the same function.)

You know the three densities on the rhs of the former equation, and your answers should be given in terms of those three densities. Now you can get any conditional density that you need using the definition and integrating to get the marginals.

For example, we have $$ p(\theta\mid y,\alpha) = \frac{p(y,\theta,\alpha)}{p(y,\alpha)} \quad , $$ where the marginal in the denominator is $$ p(y,\alpha) = \int p(y,\theta,\alpha) \,d\theta = p(\alpha) \int p(y\mid\theta,\alpha) p(\theta\mid\alpha)\, d\theta \, . $$ Therefore, we have $$ p(\theta\mid y,\alpha)=\frac{p(y\mid\theta,\alpha) \, p(\theta\mid\alpha)}{\int p(y\mid\theta,\alpha) p(\theta\mid\alpha)\, d\theta} \quad\qquad . $$ And so on.

This is a general boring way to do it, but of course, in many cases, you have shortcuts using Bayes's rule directly, etc.

As an example, the lazy way to compute $p(\alpha\mid y,\theta)$ is to notice that by Bayes's rule it is proportional to $p(y,\theta\mid \alpha)\,p(\alpha)$, which is equal, by the product rule, to $p(y\mid\theta,\alpha)\,p(\theta\mid\alpha)\,p(\alpha)$. Hence, we normalize (integrate in $\alpha$) and find $$ p(\alpha\mid y,\theta) = \frac{p(y\mid\theta,\alpha)\,p(\theta\mid\alpha)\,p(\alpha)}{\int p(y\mid\theta,\alpha)\,p(\theta\mid\alpha)\,p(\alpha)\,d\alpha} \quad\qquad . $$

Answer 2

Note that you have the joint pdf factored into three parts:

$$p(y\alpha\theta)=p(\alpha)p(\theta|\alpha)p(y|\alpha\theta)$$

Once you have this all other distributions follow from marginalising this joint distribution and the rule for conditional probability.