5
$\begingroup$

How to get the marginal pdf of $p(y)$? Do you just integrate out $p({\sigma}^{2})$?

Say, the following joint distribution for $y \in {{R}^{d}}$ and ${{\sigma }^{2}}\in {{R}^{d}}$ IG: means inverse Gamma $${{\sigma }^{2}}\sim IG(\alpha ,\beta )\propto {{({{\sigma }^{2}})}^{-(\alpha +1)}}{{e}^{-\beta /{{\sigma }^{2}}}}$$

$$y|{{\sigma }^{2}}\sim N(\mu ,{{\sigma }^{2}}\Sigma )$$

where $a\in R$, $b\in R$,$\mu \in {{R}^{d}}$,$\Sigma \in {{R}^{d\times d}}$ are known parameters.

I know that $$p(y|\mu,\Sigma)\propto \frac{1}{{{\left| {{\sigma }^{2}}\Sigma \right|}^{\frac{n}{2}}}}\exp \left[ -\frac{1}{2}\sum\limits_{i=1}^{n}{{{\left( {{y}_{i}}-\mu \right)}^{T}}{{({{\sigma }^{2}}\Sigma )}^{-1}}({{y}_{i}}-\mu )} \right]$$ $$\propto \frac{1}{{{\left| {{\sigma }^{2}}\Sigma \right|}^{\frac{n}{2}}}}\exp \left[ -\frac{1}{2}\sum\limits_{i=1}^{n}{{{\left( {{y}_{i}}-\bar{y} \right)}^{T}}{{({{\sigma }^{2}}\Sigma )}^{-1}}({{y}_{i}}-\bar{y})-\frac{n}{2}{{(\bar{y}-\mu )}^{T}}{{({{\sigma }^{2}}\Sigma )}^{-1}}(\bar{y}-\mu )} \right]$$ $$\propto \frac{1}{{{\left| {{\sigma }^{2}}\Sigma \right|}^{\frac{n}{2}}}}\exp \left[ -\frac{n-1}{2}tr\left( {{({{\sigma }^{2}}\Sigma )}^{-1}}S \right)-\frac{n}{2}{{(\bar{y}-\mu )}^{T}}{{({{\sigma }^{2}}\Sigma )}^{-1}}(\bar{y}-\mu ) \right]$$

what I have done is using

$$p{({\sigma}^{2})}{\times}p(y|{{\sigma }^{2}})$$

which gives me, $$\propto {{\left( {{\sigma }^{2}} \right)}^{-(\alpha +1)}}{{e}^{-\beta /{{\sigma }^{2}}}}\times \frac{1}{{{\left| {{\sigma }^{2}}\Sigma \right|}^{\frac{n}{2}}}}\exp \left[ -\frac{n-1}{2}tr\left( {{({{\sigma }^{2}}\Sigma )}^{-1}}S \right)-\frac{n}{2}{{(\bar{y}-\mu )}^{T}}{{({{\sigma }^{2}}\Sigma )}^{-1}}(\bar{y}-\mu ) \right]$$

This doesn't look like anything to me, am I even on the right track?! Anyways, if any experts knows, please point me out, thanks sooo much!

$\endgroup$
  • 1
    $\begingroup$ Is this homework? If so, please add the "homework" tag... we'll still help guide you to an answer (rather than just giving it to you.) $\endgroup$ – jbowman Feb 20 '12 at 18:24
  • $\begingroup$ The answer is no! $\endgroup$ – Xi'an Feb 20 '12 at 19:11
  • $\begingroup$ More seriously, your last line is completely correct. So you are on the right track: remember that only $\sigma^2$ varies in this expression and that you can factorise most of the term in the exponential in $\sigma^2$. Then you should see a standard distribution in $\sigma^2$. $\endgroup$ – Xi'an Feb 20 '12 at 21:04
  • $\begingroup$ So, I think I will integrate the joint density(my last line) w.r.t. $\sigma^{2}$, then I get p(y). Is that also what you are saying? But I am not too sure about integration matrix. I don't get what you mean by "factorise them". Something like complete the square? $\endgroup$ – user1061210 Feb 20 '12 at 21:27
  • 1
    $\begingroup$ Let me cook dinner and I will explain..! $\endgroup$ – Xi'an Feb 21 '12 at 19:19
4
$\begingroup$

What you get as your bottom line is of the form $$ (\sigma^2) ^{-\alpha-1-nd/2}\exp\{-A\sigma^{-2}\} $$ so the posterior distribution in $\sigma^{-2}$ is an inverse gamma distribution. (Note that $$ \text{tr}((\sigma^2\Sigma)^{-1}S)=\sigma^{-2}\text{tr}(\Sigma^{-1}S)\,.) $$ From this property, you can derive the normalising constant.

$\endgroup$
  • $\begingroup$ Does this look right? $p(y)=\frac{1}{{{\left| \Sigma \right|}^{n/2}}}\frac{\Gamma (a+n/2)}{{{\left( b+\frac{n-1}{2}tr\left( {{\Sigma }^{-1}}S \right)+\frac{n}{2}{{(\bar{y}-\mu )}^{T}}{{(\Sigma )}^{-1}}(\bar{y}-\mu ) \right)}^{a+n/2}}}$ $\endgroup$ – user1061210 Feb 22 '12 at 3:31
  • $\begingroup$ this is not any distribution I could recognize, more like a normalising constant with y bar in it. $\endgroup$ – user1061210 Feb 22 '12 at 3:32
  • 2
    $\begingroup$ Note that you can write the denominator as $\beta+(y-\mu)^T\Sigma^{-1}(y-\mu)$. note that the degrees of freedom is equal to double the power of this term minus the dimension of $y$ which gives us $v=2\alpha+n-d$. thus we can write posterior as $$p(y)\propto\left[1+\frac{1}{v}(y-\mu)\left(\frac{\beta}{v}\Sigma\right)^{-1}(y-\mu)\right]^{-\frac{v+d}{2}}$$ this is the kernel of a student t distribution. $\endgroup$ – probabilityislogic Feb 22 '12 at 5:09
  • $\begingroup$ Yes, this is a Student's $t$ density. $\endgroup$ – Xi'an Feb 22 '12 at 5:19
  • $\begingroup$ Apologies the above is a bit wrong. The exponent should be $\frac{v+nd}{2}$ and $v=2\alpha+n-nd$. we should also have a block diagonal matrix with $n$ blocks equal to $\Sigma$ instead of just $\Sigma$. also the term $(y-\mu)$ should be replaced by the $1\times nd$ row vector $\left[(y_1-\mu)^T,(y_2-\mu)^T,\dots,(y_n-\mu)^T\right]$. $\endgroup$ – probabilityislogic Feb 22 '12 at 5:44
2
$\begingroup$

Note that the normalising constant for a IG variable is

$$\frac{b^a}{\Gamma(a)}$$

This is equal to the reciprical of the integral over $\sigma^{2}$ of the kernel of the pdf. hence we have

$$\int_0^{\infty}(\sigma^{2})^{-(a+1)}\exp\left(-\frac{b}{\sigma^2}\right)d\sigma^2=\frac{\Gamma(a)}{b^a}$$

Your integral is of this form for certain choice of $a$ and $b$.

$\endgroup$
  • $\begingroup$ @user1061210 - the inverse gamma distribution is undefined for $\sigma^2\leq 0$. The integral is then infinite for negative range. You cannot have negative range for this parameter, as it is a normal distribution variance. Similarly $\Sigma$ must be positive semi-definite (i.e. $x^T\Sigmax\geq 0$ for any vector $x$) for your distribution for $p(y|\sigma^2)$ to be valid $\endgroup$ – probabilityislogic Feb 22 '12 at 2:45
  • $\begingroup$ Does this look good? continued from my last line $$={{\left( {{\sigma }^{2}} \right)}^{-(\alpha +1)-n/2}}\frac{1}{{{\left| \Sigma \right|}^{n/2}}}\exp \left\{ -\frac{1}{{{\sigma }^{2}}}\left[ b-\frac{n-1}{2}tr\left( {{\Sigma }^{-1}}S \right)-\frac{n}{2}{{(\bar{y}-\mu )}^{T}}{{(\Sigma )}^{-1}}(\bar{y}-\mu ) \right] \right\}$$ $\endgroup$ – user1061210 Feb 22 '12 at 3:28
  • $\begingroup$ integrate out ${sigma}^{2}$ $$=\frac{1}{{{\left| \Sigma \right|}^{n/2}}}\int_{0}^{\infty }{{{\left( {{\sigma }^{2}} \right)}^{-(a+1+n/2)}}\exp \left\{ -\frac{1}{{{\sigma }^{2}}}\left[ b+\frac{n-1}{2}tr\left( {{\Sigma }^{-1}}S \right)+\frac{n}{2}{{(\bar{y}-\mu )}^{T}}{{(\Sigma )}^{-1}}(\bar{y}-\mu ) \right] \right\}}d{{\sigma }^{2}}$$ $\endgroup$ – user1061210 Feb 22 '12 at 3:30
  • $\begingroup$ $p(y)=\frac{1}{{{\left| \Sigma \right|}^{n/2}}}\frac{\Gamma (a+n/2)}{{{\left( b+\frac{n-1}{2}tr\left( {{\Sigma }^{-1}}S \right)+\frac{n}{2}{{(\bar{y}-\mu )}^{T}}{{(\Sigma )}^{-1}}(\bar{y}-\mu ) \right)}^{a+n/2}}}$ $\endgroup$ – user1061210 Feb 22 '12 at 3:31
  • $\begingroup$ Yep you got it. Note that this is called a multivariate student distribution. It is not properly normalised though. So you should replace $=$ with $\propto$ $\endgroup$ – probabilityislogic Feb 22 '12 at 3:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.