2
$\begingroup$

My Question

Consider the model $X_1, \dots, X_n \overset{\mathrm{iid}}{\sim} \cal N(\mu, \sigma^2), (\mu, \sigma^2) \in (-\infty, \infty) \times (0, \infty)$ with a product prior pdf $\xi(\mu, \sigma^2) = \xi_1(\mu)\xi_2(\sigma^2)$ where $\xi_1(\mu) = \cal N(a, b^2)$ and $\xi_2(\sigma^2) = \chi^{-2}(r,s)$ for some constants $a \in , b > 0, r > 0$ and $s > 0$. Let $\xi(\mu, \sigma^2| x)$ denote the posterior pdf given observations $x = (x_1, \dots, x_n)$.

Identify the conditional posterior pdf $\xi_1(\mu, | x, \sigma^2)$ of $\mu$ for a given $\sigma^2$.

Identify the conditional posterior pdf $\xi_2(\sigma^2 | x, \mu)$ of $\sigma^2$ for a given $\mu$.

My Thoughts

First, a helpful fact from my previous searching: $$\xi_1(\mu | x, \sigma^2) \propto \xi(\mu, \sigma^2 | x)$$ Given this, I fixed $\mu, \sigma^2$ in turn and saw what kind of pdf I got. For $\mu$, fixing $\sigma^2$, I got another Normal pdf, so my belief is that $\xi_1(\mu, | x, \sigma^2)$ should take the form of a Normal pdf.

Fixing $\mu$, I saw that $\sigma^2$ was in the denominator of the exponential term and in the denominator of the ${1\over \sigma \sqrt{2\pi}}$ term outside of the exponential term, and so I thought of the inverse-chi-square family or the inverse gamma family.

My Question

Given that I have reasonable beliefs about what form my answers should take, how would I go about finding explicit formulae for the updated parameters? I want to be able to explicitly write down a formula for $\xi_1(\mu, | x, \sigma^2)$ and right now, given that I identified the form they should take through (liberal) use of "$\propto$", I'm not sure what to do.

$\endgroup$
  • $\begingroup$ The inverse-chi-squared distribution has just one parameter $\nu$. Can you clarify the meaning of $r$ and $s$? $\endgroup$ – Zen Oct 19 '12 at 2:12
4
$\begingroup$

You have random variables $X_1,\dots,X_n$ that are, given that $M=\mu$ and $\Sigma^2=\sigma^2$, conditionally independent and identically distributed as $\mathrm{N}(\mu,\sigma^2)$.

You believe that a priori $M$ and $\Sigma^2$ are independent with $M\sim\mathrm{N}(a,b^2)$ and $\Sigma\sim\mathrm{IG}(r,t)$.

(please, check the definition of the Inverse-Gamma distribution).

Define $X=(X_1,\dots,X_n)$ and $x=(x_1,\dots,x_n)$. Also, define $\bar{x}=\sum_{i=1}^n x_i/n$ and $s^2=\sum_{i=1}^n (x_i-\bar{x})^2/n$.

The likelihood is $$ f_{X\mid M,\Sigma^2}(x\mid\mu,\sigma^2) = \prod_{i=1}^n f_{X_i\mid M,\Sigma^2}(x_i\mid\mu,\sigma^2) $$ $$ = (2\pi)^{-n/2} \sigma^{-n} \exp\left(-\frac{1}{2\sigma^2}\sum_{i=1}^n (x_i-\mu)^2\right) \, . $$ But $$ \sum_{i=1}^n (x_i-\mu)^2 = \sum_{i=1}^n (x_i-\bar{x}+\bar{x}-\mu)^2 $$ $$ = \sum_{i=1}^n (x_i-\bar{x})^2 +2 (\bar{x}-\mu)\sum_{i=1}^n (x_i-\bar{x}) + \sum_{i=1}^n (\bar{x}-\mu)^2 \, . \quad (*) $$ The middle term in $(*)$ is zero (check it out), giving us the useful decomposition $$ \sum_{i=1}^n (x_i-\mu)^2 = n(\mu-\bar{x})^2 + ns^2 \, , $$ which allows us to rewrite the likelihood in terms of the sufficient statistics $\bar{x}$ and $s^2$ as $$ f_{X\mid M,\Sigma^2}(x\mid\mu,\sigma^2) = (2\pi)^{-n/2} \sigma^{-n} \exp\left(-\frac{n}{2\sigma^2}\left((\mu-\bar{x})^2 + s^2\right)\right) \, . $$ Hence, using the most beautiful theorem ever, the posterior density is $$ f_{M,\Sigma^2\mid X}(\mu,\sigma^2\mid x) \propto f_{X\mid M,\Sigma^2}(x\mid\mu,\sigma^2) f_M(\mu) f_{\Sigma^2}(\sigma^2) $$ $$ \propto \sigma^{-n} \exp\left(-\frac{n}{2\sigma^2}\left((\mu-\bar{x})^2 + s^2\right)\right) $$ $$ \times \exp\left(-\frac{1}{2b^2}(\mu-a)^2\right) (\sigma^2)^{-r-1} \exp\left(-\frac{t}{\sigma^2}\right) \, , $$ in which the $\propto$ symbol means proportionality up to factors that do not depend on $\mu$ and $\sigma^2$.

To compute the full conditionals, note that, by the product rule for densities, we have $$ f_{M, \Sigma^2\mid X}(\mu,\sigma^2\mid x) = f_{M\mid\Sigma^2,X}(\mu\mid\sigma^2,x) f_{\Sigma^2\mid X}(\sigma^2\mid x) \, , $$ and hence $$ f_{M\mid\Sigma^2,X}(\mu\mid\sigma^2,x) \propto f_{M,\Sigma^2\mid X}(\mu,\sigma^2\mid x) \, , $$ in which the $\propto$ symbol means proportionality up to factors that do not depend on $\mu$.

Therefore, $$ f_{M\mid\Sigma^2,X}(\mu\mid\sigma^2,x) \propto \exp\left(-\frac{1}{2}\left(\frac{n}{\sigma^2}(\mu-\bar{x})^2 +\frac{1}{b^2}(\mu-a)^2\right)\right) \, . $$

Completing the square, we have $$ \frac{n}{\sigma^2}(\mu-\bar{x})^2 +\frac{1}{b^2}(\mu-a)^2 \propto \left(\frac{nb^2+\sigma^2}{\sigma^2 b^2}\;\;\right) \mu^2 - 2\mu\left(\frac{nb^2\bar{x}+\sigma^2a}{\sigma^2b^2}\;\;\right) $$ $$ \propto \left(\frac{nb^2+\sigma^2}{\sigma^2 b^2}\;\;\right)\left(\mu - \frac{nb^2\bar{x}+\sigma^2a}{nb^2+\sigma^2}\;\;\right)^2 \, , $$ which gives you, by inspection, that $$ M\mid\Sigma^2=\sigma^2,X=x \sim \mathrm{N}\left(\frac{nb^2\bar{x}+\sigma^2a}{nb^2+\sigma^2}\quad, \frac{\sigma^2 b^2}{nb^2+\sigma^2}\;\;\right) \, . $$

Now we take a deep breath and do it mutatis mutandis for the other conditional. $$ f_{\Sigma^2\mid M,X}(\sigma^2\mid \mu,x) \propto (\sigma^2)^{-n/2-r-1} \exp\left( -\frac{1}{\sigma^2}\left(\frac{n}{2}\left((\mu-\bar{x})^2 + s^2\right) + t \right)\right) \, , $$ which says that $$ \Sigma^2\mid M=\mu,X=x \sim \mathrm{IG}\left( \frac{n}{2}+r, \frac{n}{2}\left((\mu-\bar{x})^2 + s^2\right) + t\right) \, . $$

$\endgroup$
  • $\begingroup$ No, I won't be implementing a Gibbs sampler with this particular conditional posterior, but I was asked about it, and wanted to see what it would come out to be. I'm not turning in this work, I just wanted to see what the steps would be to get this posterior in $\mu, \sigma^2$ repectively. $\endgroup$ – Moderat Oct 19 '12 at 19:50
1
$\begingroup$

If your hyper parameters are fixed (without any random distribution), you can use the MCMC method to simulate and update your parameters. MCMC does not need exact distribution function of the posteriors (with normalization constant). It only needs some candidate distributions for your parameters that you can define based on your opinion about each posteriors.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.