I am reading the book The Elements of Statistical Learning by Hastie, Tibshirani and Friedman. On page 271 the authors derive a posterior distribution from a single observation $z\sim N(\theta, 1)$, where the prior of $\theta$ is specified as $\theta \sim N(0, \tau)$. It then follows (according to the authors) that the posterior distribution equals $\theta | z \sim N\left( \frac{z}{1+\frac{1}{\tau}}, \frac{1}{1+\frac{1}{\tau}} \right).$

Now, my calculations yield

\begin{align} \Pr\left(\theta |\textbf{Z}\right) &= \frac{\Pr\left(\textbf{Z} \mid \theta\right) \Pr(\theta)}{\int \Pr\left(\textbf{Z} \mid \theta\right) \Pr(\theta)d\theta} \propto \Pr\left(\textbf{Z} \mid \theta\right) \Pr(\theta) \\ &= \frac{1}{\sqrt{2\pi}}\exp\left(-\frac{1}{2}(z-\theta)^2 \right) \frac{1}{\sqrt{2\pi\tau}}\exp\left(-\frac{1}{2\tau}\theta^2 \right) \\ &= \frac{1}{2\pi\sqrt{\tau}}\exp\left(-\frac{1}{2} (z^2 + \theta^2 -2z\theta + \frac{\theta ^2}{\tau}) \right) \\ &= \frac{1}{2\pi\sqrt{\tau}}\exp\left(-\frac{1}{2} (\theta^2(1+\frac{1}{\tau}) + z^2 -2z\theta) \right) \\ &= \frac{1}{2\pi\sqrt{\tau}}\exp\left(-\frac{1}{2 \frac{1}{1+\frac{1}{\tau}}} (\theta^2 + \frac{z^2}{1+\frac{1}{\tau}} -2 \frac{z\theta}{1+\frac{1}{\tau}} ) \right). \end{align}

The denominator of $\frac{z^2}{1+\frac{1}{\tau}} $ should equal $(1+\frac{1}{\tau})^2$ for me to be able to "complete the square" and get

\begin{align} \Pr\left(\theta |\textbf{Z}\right) &\propto \frac{1}{2\pi\sqrt{\tau}}\exp\left(-\frac{1}{2 \frac{1}{1+\frac{1}{\tau}}} (\theta^2 + \frac{z^2}{(1+\frac{1}{\tau})^2} -2 \frac{z\theta}{1+\frac{1}{\tau}} ) \right) \\ &=\text{constant}\times\exp\left(-\frac{1}{2 \frac{1}{1+\frac{1}{\tau}}} (\theta - \frac{z}{1+\frac{1}{\tau}})^2 \right), \end{align}

such that $\theta | z \sim N\left( \frac{z}{1+\frac{1}{\tau}}, \frac{1}{1+\frac{1}{\tau}} \right)$.

My question is:

Where do I go wrong in the process? Should I divide with $\int \Pr\left(\textbf{Z} \mid \theta\right) \Pr(\theta)d\theta = \Pr(\textbf{Z})$? If so, what is the difference between $\Pr(\textbf{Z})$ and $\Pr\left(\textbf{Z} \mid \theta\right)$ in this given example?

Best regards,



Since we're looking for the pdf of $\theta$, we're only concerned with terms that include it.

\begin{align} \Pr\left(\theta |\textbf{Z}\right) &\propto \Pr\left(\textbf{Z} \mid \theta\right) \Pr(\theta) \\ &\propto \exp\left(-\frac{1}{2}(z-\theta)^2 -\frac{1}{2\tau}\theta^2 \right) \\ &= \exp\left(-\frac{1}{2}\left((1+\frac{1}{\tau})\theta^2 -2z\theta+z^2 \right)\right)\\ &= \exp\left(-\frac{1}{2}(1+\frac{1}{\tau})\left(\theta^2 -2\frac{z}{1+\frac{1}{\tau}}\theta+\frac{z^2}{1+\frac{1}{\tau}} \right)\right)\\ &\propto \exp\left(-\frac{1}{2}(1+\frac{1}{\tau})\left(\theta - \frac{z}{1+\frac{1}{\tau}} \right)^2\right) \end{align} And that last line implies the desired result.

  • 1
    $\begingroup$ Ah yes, of course, thanks a lot and merry christmas :) $\endgroup$ – wanderingashenvalewisp Dec 21 '20 at 17:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.