4
$\begingroup$

I am trying to understand how to derive the posterior distribution of a parameter $\mu$ given data vector $z$, $P(\mu|z)$, where $$ \mu \sim N(0,A) $$ and $$ z|\mu \sim N(\mu,1). $$

Obviously from Bayes theorem $$ P(\mu|z) = g(\mu) f(z|\mu) /f(z). $$ where $$ f(z)= \int g(\mu) f(z|\mu) d \mu . $$

I can write $g(\mu) f(z|\mu)$ as a pointwise product of normal densities:

$$ \frac{1}{2 \pi \sqrt{A}} \exp \bigg(-\frac{\mu^2+A(z-\mu)^2}{2A}\bigg) $$

the solution is given in Efron's book on Large Scale Inference, p. 2. here:

$$ \mu|z \sim N(zB,B)$$ where $$B=\frac{A}{A+1} $$

I would appreciate advice on how to approach the problem (and the answer to this question is a proof). In particular I do not understand what to do with the integral in the numerator of Bayes theorem.

EDIT Following answer by @Neil_G I took a next approach:

We have: \begin{align} g(\mu) &\propto \exp\bigg(- \frac{\mu^2}{2A}\bigg) \\[8pt] f(z|\mu) &\propto \exp\bigg(z \mu - \frac{1}{2}(z^2+\mu^2)\bigg) \end{align}

so \begin{align} P(\mu|z) &\propto \exp\bigg(- \frac{1}{2A} \mu^2 + z\mu - \frac{1}{2} z^2 - \frac{1}{2} \mu^2\bigg) \\[8pt] &= \exp \bigg(-\frac{1}{2B} (\mu^2+Bz^2-2Bz \mu)\bigg) \\[8pt] &\propto \exp \bigg(-\frac{1}{2B} (\mu^2-2Bz \mu)\bigg) \\[8pt] &\propto \exp \bigg(-\frac{ (\mu - Bz)^2}{2B}\bigg) \end{align}

which completes the proof.

$\endgroup$
7
  • 1
    $\begingroup$ This will be much easier if you write $\mu$ in the the natural parametrization (mean times precision and precision) so that your updates to it are addition. $\endgroup$
    – Neil G
    Commented Dec 6, 2015 at 17:54
  • $\begingroup$ Also forget about $f(z)$ — just write Bayes rule with $\propto$ $\endgroup$
    – Neil G
    Commented Dec 6, 2015 at 17:55
  • $\begingroup$ What do you mean by natural parameterization? $\endgroup$
    – tomka
    Commented Dec 6, 2015 at 17:56
  • $\begingroup$ it's the parametrization $a, b$ such that a normal density can be written $f(x \mid a, b) \propto \exp(ax + bx^2)$. Therefore, the pointwise product of densities having parameters $a_1, b_1$ and $a_2, b_2$ is simply the sum $(a_1+ a_2, b_1+b_2)$. $\endgroup$
    – Neil G
    Commented Dec 6, 2015 at 17:58
  • $\begingroup$ Complete the square w.r.t $\mu$ to find the solution. $\endgroup$
    – aphe
    Commented Dec 6, 2015 at 18:04

2 Answers 2

3
$\begingroup$

First note that \begin{align} x \sim N(\mu, \sigma^2) &\Leftrightarrow f(x \mid \mu, \sigma^2) \propto \exp\left(\frac\mu{\sigma^2}x -\frac1{2\sigma^2}x^2\right) & \tag{1} \\ x \sim N(\mu, \sigma^2) &\Rightarrow L(\mu \mid x, \sigma^2) \propto \exp\left(\frac{x}{\sigma^2}\mu -\frac1{2\sigma^2}\mu^2\right) & \tag{2} \end{align}

So, \begin{align} P(\mu) &\propto \exp\left(-\frac{1}{2A}\mu^2\right) & \text{by (1)} \\ L(\mu \mid z) &\propto \exp\left(\mu z - \frac12\mu^2\right) & \text{by (2)} \\ \implies P(\mu \mid z) &\propto P(\mu) L(\mu \mid z) \\ &\propto \exp\left(\mu z - \frac{A + 1}{2A}\mu^2\right) \\ \implies \mu &\sim N\left(z\frac{A}{A+1}, \frac{A}{A+1}\right) & \text{by (1)}. \end{align}

$\endgroup$
10
  • $\begingroup$ (Not sure if I made any mistakes, but this is how I would go about it.) $\endgroup$
    – Neil G
    Commented Dec 6, 2015 at 18:33
  • $\begingroup$ in the second line, don't you forget $-1/2 z^2$? $\endgroup$
    – tomka
    Commented Dec 6, 2015 at 18:47
  • $\begingroup$ @ThomasKlausch That's a constant, so we can forget about it. $\endgroup$
    – Neil G
    Commented Dec 6, 2015 at 18:48
  • $\begingroup$ Thanks, can you check the edit to my post? $\endgroup$
    – tomka
    Commented Dec 6, 2015 at 19:17
  • 1
    $\begingroup$ @ThomasKlausch: Glad it makes sense :) I think that's it, yeah. $\endgroup$
    – Neil G
    Commented Dec 6, 2015 at 19:27
2
$\begingroup$

The normal prior for the mean of a normal model represents a Conjugate Prior:

https://en.wikipedia.org/wiki/Conjugate_prior

This implies that the posterior of $\mu$ is also normal with certain parameters. This is a classical exercise in any introductory course in Bayesian statistics. The trick consists of expanding the binomial and then factorising in terms of $\mu$ in order to retrieve the normal kernel. Take a look, for instance, at the following lecture notes

http://www.cs.berkeley.edu/~jordan/courses/260-spring10/lectures/lecture5.pdf

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.