This is a problem from a machine learning pset that I'm self-learning from http://www.seas.harvard.edu/courses/cs281/assignment-1.pdf.

Suppose we are provided with a hierarchy of three distributions: $p(\alpha)$, $p(\theta | \alpha)$ and $p(y | \theta, \alpha)$. Write expressions to compute the following related distributions:

$p(y|\theta)$, $p(y|\alpha)$, $p(y)$, $p(\theta|y,\alpha)$, $p(\alpha|y,\theta)$, $p(\theta)$, $p(\theta|y)$, $p(\alpha|y)$

I see that I integrate over the joint distribution to get the marginal probability

$$p(y|\theta) = \int p(y|\theta,\alpha)p(\alpha)d\alpha$$

This gives me the first expression in terms of the givens.

However, I don't know how to get the others. I know I can set up various equations using Bayes' Theorem

$$p(\theta|\alpha) = p(\alpha|\theta) * p(\theta)/p(\alpha)$$

But the above equation has two unknowns ($p(\theta)$ and $p(\alpha|\theta)$). Is there another property that I can exploit besides marginalizing the joint distribution and utilizing Bayes' Theorem?

  • $\begingroup$ Hint: You can find $p(\theta) = \int p(\theta\mid\alpha)p(\alpha)d\alpha$ and use this to get $p(y\mid\alpha)$ from $p(y\mid\theta,\alpha)$ $\endgroup$ – Dilip Sarwate Oct 8 '12 at 2:53
  • $\begingroup$ Your expression for $p(y|\theta)$ is wrong. It should be $$\int p(y|\theta\alpha)p(\alpha|\theta)d\alpha=\frac{\int p(y|\theta\alpha)p(\alpha)p(\theta|\alpha)d\alpha}{\int\int p(y|\theta\alpha)p(\alpha)p(\theta|\alpha)d\alpha dy}$$ $\endgroup$ – probabilityislogic Oct 8 '12 at 4:48
  • $\begingroup$ Just realised you can remove $p(y|\theta\alpha)$ and the integration over $y$ in the denominator of my answer, as $\int p(y|\alpha\theta)dy=1$ $\endgroup$ – probabilityislogic Oct 8 '12 at 5:07
  • $\begingroup$ @probabilityislogic thanks so much for pointing that out! Yes, I think the correct expression should have been $p(y|\theta) = \int p(y|\theta,\alpha)p(\alpha|\theta)d\alpha$ which contains an unknown $\endgroup$ – gjx Oct 8 '12 at 16:06

The given conditionals allow you to find the joint density of the three random variables using the product rule $$ p(y,\theta,\alpha)=p(y\mid\theta,\alpha) \, p(\theta\mid\alpha) \, p(\alpha) \, . $$

(Of course this is a sloppy notation, since the $p$'s are not the same function.)

You know the three densities on the rhs of the former equation, and your answers should be given in terms of those three densities. Now you can get any conditional density that you need using the definition and integrating to get the marginals.

For example, we have $$ p(\theta\mid y,\alpha) = \frac{p(y,\theta,\alpha)}{p(y,\alpha)} \quad , $$ where the marginal in the denominator is $$ p(y,\alpha) = \int p(y,\theta,\alpha) \,d\theta = p(\alpha) \int p(y\mid\theta,\alpha) p(\theta\mid\alpha)\, d\theta \, . $$ Therefore, we have $$ p(\theta\mid y,\alpha)=\frac{p(y\mid\theta,\alpha) \, p(\theta\mid\alpha)}{\int p(y\mid\theta,\alpha) p(\theta\mid\alpha)\, d\theta} \quad\qquad . $$ And so on.

This is a general boring way to do it, but of course, in many cases, you have shortcuts using Bayes's rule directly, etc.

As an example, the lazy way to compute $p(\alpha\mid y,\theta)$ is to notice that by Bayes's rule it is proportional to $p(y,\theta\mid \alpha)\,p(\alpha)$, which is equal, by the product rule, to $p(y\mid\theta,\alpha)\,p(\theta\mid\alpha)\,p(\alpha)$. Hence, we normalize (integrate in $\alpha$) and find $$ p(\alpha\mid y,\theta) = \frac{p(y\mid\theta,\alpha)\,p(\theta\mid\alpha)\,p(\alpha)}{\int p(y\mid\theta,\alpha)\,p(\theta\mid\alpha)\,p(\alpha)\,d\alpha} \quad\qquad . $$


Note that you have the joint pdf factored into three parts:


Once you have this all other distributions follow from marginalising this joint distribution and the rule for conditional probability.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.