2
$\begingroup$

Find the joint posterior of $(\mu, \sigma^2)$ given Normal data.

I've found the joint prior of $\mu$ and $\sigma^2$ (where $\displaystyle\sigma^2\sim\chi^{-2}(v_o,v_os_o^2)$ and $\mu\mid\sigma^2\sim N(\mu_o,\frac{\sigma^2}{n_o})$) and likelihood function:

$$\displaystyle\pi(\mu,\sigma^2)\propto(\sigma^2)^{-\frac{v_o+1}{2}-1}e^{-\frac{1}{2\sigma^2}[v_os_o^2+n_o(\mu-\mu_o)^2]}$$ and $$L(\mu,\sigma^2\mid x_1,\ldots,x_n)\propto (\sigma^2)^{-\frac{n}{2}}e^{-\frac{1}{2\sigma^2}[(n-1)s^2+n(\overline x-\mu)^2]}$$ respectively.

So the joint posterior of $\mu$ and $\sigma^2$ given the data is proportional to the product of both

$$\displaystyle (\sigma^2)^{-\frac{v_o+1}{2}-1}e^{-\frac{1}{2\sigma^2}[v_os_o^2+n_o(\mu-\mu_o)^2]} (\sigma^2)^{-\frac{n}{2}}e^{-\frac{1}{2\sigma^2}[(n-1)s^2+n(\overline x-\mu)^2]}$$

We also know that $$\displaystyle\pi(\mu,\sigma^2)\propto(\sigma^2)^{-\frac{v_o+1}{2}-1}e^{-\frac{1}{2\sigma^2}[v_os_o^2+n_o(\mu-\mu_o)^2]}$$ is an inverse Gamma normal distribution with parameters $(v_o,s_o^2,\mu_o,n_o).$

I am expected to identify the joint posterior of $\mu$ and $\sigma^2$ given the data as an inverse Gamma-Normal distribution with similar parameters to $(v_o,s_o^2,\mu_o,n_o)$.

My question is which are the new parameters and what 'tricks' will I need to apply so that it has the form of inverse Gamma normal distribution?


Update

Checking the answer given,

  • How Xi'an passed from $(\sigma^2)^{-\frac{v_o+n}{2}-1}e^{-\frac{1}{2\sigma^2}[v_os_o^2+(n-1)s^2+n_o\mu_o^2+n \overline x^2-(n_o\mu_o+n\overline x)^2/(n_o+n)]}$ to $(\sigma^2)^{-\frac{v_o+n}{2}-1}e^{-\frac{1}{2\sigma^2}[v_os_o^2+(n-1)s^2+n_ov_o(\overline x-\mu_o)^2/(n_o+v_o)^2]}$ ?

Notice that only this $n_o\mu_o^2+n \overline x^2-(n_o\mu_o+n\overline x)^2/(n_o+n)$ changed to $n_ov_o(\overline x-\mu_o)^2/(n_o+v_o)^2$ though the term $v_o$ is whether a typo or what is representing? Because it's a new term in the second expression that does not exist in the first expression.

  • How can this be represented using R ? How can I plot the prior and posterior distributions, for a certain given parameters?
$\endgroup$
  • 3
    $\begingroup$ There's no trick unless you think "collect terms and simplify" and "equate coefficients of corresponding terms" are tricks. $\endgroup$ – Glen_b Jan 13 '19 at 6:36
  • $\begingroup$ @Xi'an I'll edit later about that part. $\endgroup$ – user208618 Jan 27 '19 at 19:45
  • $\begingroup$ @Xi'an Regarding to the first part of your answer, you say that $\pi(\mu|\sigma^2)$ is proportional to an inverse Gamma with parameters $\frac{1}{2}(v_o+n+1,v_os_o^2+n_o(\mu-\mu_o)^2+(n-1)s^2+n(\overline x-\mu)^2)$. From here I can see the first parameter $\alpha$ but not $\beta$. More specifically, how can I rewrite $n_o(\mu-\mu_o)^2+n(\overline x-\mu)^2$ so that have the form $n_1(\mu-\mu_1) ?$ $\endgroup$ – user208618 Jan 27 '19 at 19:46
  • $\begingroup$ @Xi'an I think I did it, I got $(n_o+n)(\mu-\frac{n_o\mu_o+\overline xn}{n_o+n})^2+constant$. Is it correct? $\endgroup$ – user208618 Jan 27 '19 at 20:29
  • $\begingroup$ The first part of my answer is about $\pi(\sigma^2|\mu)$, not the reverse. And I do not exhibit a perfect square either. The explanation for $\pi(\sigma^2|\mu)$ is found in the second part and I explain there how to exhibit a perfect square for $\mu$. Which indeed leads to$$\frac{n_o+n}{2\sigma^2}[\mu -(n_o\mu_o+n\overline x)/(n_o+n)]^2$$ $\endgroup$ – Xi'an Jan 27 '19 at 21:21
4
+100
$\begingroup$

...the joint posterior of $\mu$ and $\sigma^2$ given the data is proportional to the product of both $$\displaystyle (\sigma^2)^{-\frac{v_o+1}{2}-1}e^{-\frac{1}{2\sigma^2}[v_os_o^2+n_o(\mu-\mu_o)^2]} (\sigma^2)^{-\frac{n}{2}}e^{-\frac{1}{2\sigma^2}[(n-1)s^2+n(\overline x-\mu)^2]}$$

which simplifies into $$\displaystyle \pi(\mu,\sigma^2\mid\overline{x},s)\propto(\sigma^2)^{-\frac{v_o+n+1}{2}-1}e^{-\frac{1}{2\sigma^2}[v_os_o^2+n_o(\mu-\mu_o)^2+(n-1)s^2+n(\overline x-\mu)^2]}\,.\tag{1}$$ As a function of $\sigma^2$ only (which means conditional on everything else) this expression is proportional to an inverse Gamma conditional posterior density, with parameters $$\frac{1}{2}(v_o+n+1,v_os_o^2+n_o(\mu-\mu_o)^2+(n-1)s^2+n(\overline x-\mu)^2).$$ Proportional means that the above expression is missing a multiplicative constant to turn it a probability density, with integral equal to one, a constant that is called a normalisation term or a normalising constant. The full inverse Gamma density is indeed \begin{align}\Gamma(\frac{v_o+n+1}{2})^{-1}&\times\left\{\frac{[v_os_o^2+n_o(\mu-\mu_o)^2+(n-1)s^2+n(\overline x-\mu)^2]}{2}\right\}^{\frac{v_o+n+1}{2}}\\ &(\sigma^2)^{-\frac{v_o+n+1}{2}-1}e^{-\frac{1}{2\sigma^2}[v_os_o^2+n_o(\mu-\mu_o)^2+(n-1)s^2+n(\overline{x}-\mu)^2]} \end{align} Integrating the rhs of (1) with respect to $\sigma^2$ thus produces the missing constant, $$\Gamma(\frac{v_o+n+1}{2})\times\left\{\frac{[v_os_o^2+n_o(\mu-\mu_o)^2+(n-1)s^2+n(\overline x-\mu)^2]}{2}\right\}^{-\frac{v_o+n+1}{2}}\tag{2}$$ and since integrating the joint posterior on $(\mu,\sigma^2)$ in $\sigma^2$ returns the marginal posterior density of $\mu$, $$\pi(\mu\mid\overline{x},s) = \int_0^\infty \pi(\mu,\sigma^2\mid\overline{x},s)\,\text{d}\sigma^2$$ this integral (2) is therefore the marginal posterior density of $\mu$, up to a multiplicative constant. With a wee bit of further work, (2) turns into a Student's $t$ density on $\mu$, with $\frac{v_o+n}{2}$ degrees of freedom, as also described in our book, Bayesian Essentials [pages 30-31].

Conversely, if integrating $\mu$ first from (1), the expression becomes proportional to $\pi(\sigma^2\mid\overline{x},s)$: since \begin{align}\pi(\mu,\sigma^2\mid\overline{x},s) &\propto (\sigma^2)^{-\frac{v_o+n+1}{2}-1}e^{-\frac{1}{2\sigma^2}[v_os_o^2+n_o(\mu-\mu_o)^2+(n-1)s^2+n(\overline x-\mu)^2]}\\ &=(\sigma^2)^{-\frac{v_o+n}{2}-1}e^{-\frac{1}{2\sigma^2}[v_os_o^2+(n-1)s^2+n_o\mu_o^2+n\overline{x}^2]}\overbrace{\sigma^{-1}e^{-\frac{1}{2\sigma^2}[(n_o+n)\mu^2 -2\mu(n_o\mu_o+n\overline x)]}}^\text{incomplete normal density}\\ &=(\sigma^2)^{-\frac{v_o+n}{2}-1}e^{-\frac{1}{2\sigma^2}[v_os_o^2+(n-1)s^2+n_o\mu_o^2+n\overline{x}^2]}\underbrace{\sigma^{-1}e^{-\frac{n_o+n}{2\sigma^2}[\mu -(n_o\mu_o+n\overline x)/(n_o+n)]^2}}_\text{conditional posterior normal density in $\mu$}\\ &\qquad\qquad\times \underbrace{e^{\frac{1}{2\sigma^2}[(n_o\mu_o+n\overline x)^2/(n_o+n)]}}_\text{missing term in perfect square}\end{align} integrating $\mu$ out returns \begin{align}\pi(\sigma^2\mid\mathbf{x}) &\propto (\sigma^2)^{-\frac{v_o+n}{2}-1}e^{-\frac{1}{2\sigma^2}[v_os_o^2+(n-1)s^2+n_o\mu_o^2+n\overline{x}^2-(n_o\mu_o+n\overline x)^2/(n_o+n)]}\\ &=(\sigma^2)^{-\frac{v_o+n}{2}-1}e^{-\frac{1}{2\sigma^2}[v_os_o^2+(n-1)s^2+n_ov_o(\overline x-\mu_o)^2/(n_o+v_o)^2]} \end{align} since \begin{align}n_o\mu_o^2+n\overline{x}^2-(n_o\mu_o+n\overline x)^2/(n_o+n) &=n_o\mu_o^2+n\overline{x}^2-\frac{n_o^2}{n_o+n}\mu_o^2-\frac{2n_on}{n_o+n}\mu_o\overline x-\frac{n^2}{n_o+n}\overline{x}^2\\ &= \frac{n_o\mu_o^2}{n_o+n}(n_o+n-n_o)-\frac{2n_on\mu_o\overline x}{n_o+n}+\frac{n\overline{x}^2}{n_o+n}(n_o+n-n)\\ &=\frac{nn_o}{n_o+n}(\mu_o-\overline x)^2 \end{align} which means that the marginal posterior on $\sigma^2$ is an inverse Gamma with parameters $$\frac{1}{2}(v_o+n,[v_os_o^2+(n-1)s^2+n_on(\overline x-\mu_o)^2/(n_o+n)])$$ This also follows from observing that, since $$\overline{x}\mid\mu,\sigma\sim\mathcal{N}(\mu,n^{-1}\sigma^2)\qquad \mu\mid\sigma\sim\mathcal{N}(\mu_o,n_o^{-1}\sigma^2)$$ then by marginalising in $\mu$ $$\overline{x}\mid\sigma\sim\mathcal{N}(\mu_o,[n^{-1}+n_o^{-1}]\sigma^2)$$

$\endgroup$
  • $\begingroup$ Thank you, I'm getting the point :) In your last paragraph what do you mean with 'up to a multiplicative constant.' ? $\endgroup$ – user208618 Jan 14 '19 at 21:40
  • $\begingroup$ How did you integrate $(1)$, did you use Mathematica or something else? $\endgroup$ – user208618 Jan 14 '19 at 21:46
  • $\begingroup$ And so the other term $(\sigma^2)^{-\frac{v_o+n}{2}}e^{-\frac{1}{2\sigma^2}[v_os_o^2+(n-1)s+n_o\mu_o^2+n s^2]} e^{\frac{1}{2\sigma^2}[(n_o\mu_o+n\overline x)^2/(n_o+n)]}$ is the marginal posterior of $\sigma^2$ right (I just want to be sure)? $\endgroup$ – user208618 Jan 15 '19 at 20:24
  • 1
    $\begingroup$ I was checking all again and I think there's a missing parameter in the Gamma function $\Gamma(\frac{v_o+n+1}{2})$ ? also in the Conversely part the $\sigma^{-1}$ should not be $\sigma^{-3}$ ? $\endgroup$ – user208618 Jan 17 '19 at 20:57
  • 1
    $\begingroup$ How did you passed from here $(\sigma^2)^{-\frac{v_o+n}{2}-1}e^{-\frac{1}{2\sigma^2}[v_os_o^2+(n-1)s^2+n_o\mu_o^2+n \overline x^2-(n_o\mu_o+n\overline x)^2/(n_o+n)]}$ to $(\sigma^2)^{-\frac{v_o+n}{2}-1}e^{-\frac{1}{2\sigma^2}[v_os_o^2+(n-1)s^2+n_ov_o(\overline x-\mu_o)^2/(n_o+v_o)^2]}$ ? I could notice the only that changed from one to the other is $n_o\mu_o^2+n \overline x^2-(n_o\mu_o+n\overline x)^2/(n_o+n)$ to $n_ov_o(\overline x-\mu_o)^2/(n_o+v_o)^2$ though $v_o$ Is not in the first term. Also notice I've changed $s^2$ by $\overline x^2$ (I think it's a minor typo) $\endgroup$ – user208618 Jan 24 '19 at 18:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.