4
$\begingroup$

I am trying to understand how to derive the posterior distribution of a parameter $\mu$ given data vector $z$, $P(\mu|z)$, where $$ \mu \sim N(0,A) $$ and $$ z|\mu \sim N(\mu,1). $$

Obviously from Bayes theorem $$ P(\mu|z) = g(\mu) f(z|\mu) /f(z). $$ where $$ f(z)= \int g(\mu) f(z|\mu) d \mu . $$

I can write $g(\mu) f(z|\mu)$ as a pointwise product of normal densities:

$$ \frac{1}{2 \pi \sqrt{A}} \exp \bigg(-\frac{\mu^2+A(z-\mu)^2}{2A}\bigg) $$

the solution is given in Efron's book on Large Scale Inference, p. 2. here:

$$ \mu|z \sim N(zB,B)$$ where $$B=\frac{A}{A+1} $$

I would appreciate advice on how to approach the problem (and the answer to this question is a proof). In particular I do not understand what to do with the integral in the numerator of Bayes theorem.

EDIT Following answer by @Neil_G I took a next approach:

We have: \begin{align} g(\mu) &\propto \exp\bigg(- \frac{\mu^2}{2A}\bigg) \\[8pt] f(z|\mu) &\propto \exp\bigg(z \mu - \frac{1}{2}(z^2+\mu^2)\bigg) \end{align}

so \begin{align} P(\mu|z) &\propto \exp\bigg(- \frac{1}{2A} \mu^2 + z\mu - \frac{1}{2} z^2 - \frac{1}{2} \mu^2\bigg) \\[8pt] &= \exp \bigg(-\frac{1}{2B} (\mu^2+Bz^2-2Bz \mu)\bigg) \\[8pt] &\propto \exp \bigg(-\frac{1}{2B} (\mu^2-2Bz \mu)\bigg) \\[8pt] &\propto \exp \bigg(-\frac{ (\mu - Bz)^2}{2B}\bigg) \end{align}

which completes the proof.

$\endgroup$
  • 1
    $\begingroup$ This will be much easier if you write $\mu$ in the the natural parametrization (mean times precision and precision) so that your updates to it are addition. $\endgroup$ – Neil G Dec 6 '15 at 17:54
  • $\begingroup$ Also forget about $f(z)$ — just write Bayes rule with $\propto$ $\endgroup$ – Neil G Dec 6 '15 at 17:55
  • $\begingroup$ What do you mean by natural parameterization? $\endgroup$ – tomka Dec 6 '15 at 17:56
  • $\begingroup$ it's the parametrization $a, b$ such that a normal density can be written $f(x \mid a, b) \propto \exp(ax + bx^2)$. Therefore, the pointwise product of densities having parameters $a_1, b_1$ and $a_2, b_2$ is simply the sum $(a_1+ a_2, b_1+b_2)$. $\endgroup$ – Neil G Dec 6 '15 at 17:58
  • $\begingroup$ Complete the square w.r.t $\mu$ to find the solution. $\endgroup$ – hard2fathom Dec 6 '15 at 18:04
3
$\begingroup$

First note that \begin{align} x \sim N(\mu, \sigma^2) &\Leftrightarrow f(x \mid \mu, \sigma^2) \propto \exp\left(\frac\mu{\sigma^2}x -\frac1{2\sigma^2}x^2\right) & \tag{1} \\ x \sim N(\mu, \sigma^2) &\Rightarrow L(\mu \mid x, \sigma^2) \propto \exp\left(\frac{x}{\sigma^2}\mu -\frac1{2\sigma^2}\mu^2\right) & \tag{2} \end{align}

So, \begin{align} P(\mu) &\propto \exp\left(-\frac{1}{2A}\mu^2\right) & \text{by (1)} \\ L(\mu \mid z) &\propto \exp\left(\mu z - \frac12\mu^2\right) & \text{by (2)} \\ \implies P(\mu \mid z) &\propto P(\mu) L(\mu \mid z) \\ &\propto \exp\left(\mu z - \frac{A + 1}{2A}\mu^2\right) \\ \implies \mu &\sim N\left(z\frac{A}{A+1}, \frac{A}{A+1}\right) & \text{by (1)}. \end{align}

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ (Not sure if I made any mistakes, but this is how I would go about it.) $\endgroup$ – Neil G Dec 6 '15 at 18:33
  • $\begingroup$ in the second line, don't you forget $-1/2 z^2$? $\endgroup$ – tomka Dec 6 '15 at 18:47
  • $\begingroup$ @ThomasKlausch That's a constant, so we can forget about it. $\endgroup$ – Neil G Dec 6 '15 at 18:48
  • $\begingroup$ Thanks, can you check the edit to my post? $\endgroup$ – tomka Dec 6 '15 at 19:17
  • 1
    $\begingroup$ @ThomasKlausch: Glad it makes sense :) I think that's it, yeah. $\endgroup$ – Neil G Dec 6 '15 at 19:27
2
$\begingroup$

The normal prior for the mean of a normal model represents a Conjugate Prior:

https://en.wikipedia.org/wiki/Conjugate_prior

This implies that the posterior of $\mu$ is also normal with certain parameters. This is a classical exercise in any introductory course in Bayesian statistics. The trick consists of expanding the binomial and then factorising in terms of $\mu$ in order to retrieve the normal kernel. Take a look, for instance, at the following lecture notes

http://www.cs.berkeley.edu/~jordan/courses/260-spring10/lectures/lecture5.pdf

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.