1
$\begingroup$

On these course notes, we are given the distribution of the posterior distribution https://people.eecs.berkeley.edu/~jordan/courses/260-spring10/lectures/lecture5.pdf

enter image description here

This famous result can be found in many other places, such as https://www.cs.ubc.ca/~murphyk/Papers/bayesGauss.pdf. However, I am confused about the coefficient (i.e., the $\dfrac{1}{\sqrt{2\pi}\sigma}$) term associated with the Gaussian for the posterior distribution. Here is what I am having trouble with.


We know that given $\mathcal{D} = (x_1, \ldots, x_n), x_i$ iid,

$$p(\mu|\mathcal{D}) \propto p(D|\mu)p(\mu)$$

Suppose that $$p(\mu) = \dfrac{1}{\sqrt{2\pi}\sigma_o} \exp{\dfrac{(x - \mu_o)^2}{2\sigma_o^2}}.$$ and $$p(D|\mu)= \dfrac{1}{(2\pi\sigma^2)^\frac{N}{2}} \exp(-\dfrac{1}{2\sigma^2}\sum\limits_{n = 1}^N (x_n - \mu))$$

then multiplying the expressions together, we obtain:

$$\dfrac{1}{(2\pi\sigma^2)^\frac{n}{2}} \exp{\left[-\dfrac{1}{2\sigma^2}\sum\limits_{n = 1}^N (x_n - \mu)\right]} \dfrac{1}{\sqrt{2\pi}\sigma_o} \exp{\left[\dfrac{(x - \mu_o)^2}{2\sigma_o^2}\right]}$$

While we can perform a complete the square inside of the exponential, what about the constants $\dfrac{1}{(2\pi\sigma^2)^\frac{n}{2}}$ and $\dfrac{1}{\sqrt{2\pi}\sigma_o}$?

I don't see how $\dfrac{1}{(2\pi\sigma^2)^\frac{n}{2}} *\dfrac{1}{\sqrt{2\pi}\sigma_o} = \dfrac{1}{\sqrt{2\pi}\sigma_n^2}$

where $\sigma_n^2 = (1/\sigma_o^2 + n/\sigma^2)^{-1}$ as shown in Lemma 6.

I tried playing around with the terms but I couldn't make them equal, even when $n = 1$. Have I made a mistake or is this because $p(\mu|\mathcal{D})$ is not a "true" probability distribution (i.e., doesn't integrate to 1)?

If so, how would people deal with this leading coefficient during simulation?

Addendum (see comment)

Bishop Pattern Reconigition and Machine Learning (2006) Pg. 98

enter image description here

$\endgroup$
  • 2
    $\begingroup$ It's just a constant of integration. Note that symbol "$\propto$" in your initial expression; it means "proportional to", as in "don't worry about the constant of integration until you're done with everything else, then it's whatever is needed to make the posterior integrate to one". $\endgroup$ – jbowman Jan 26 '18 at 4:25
  • $\begingroup$ It is $1/\sqrt{2\pi}\sigma_n$. $\endgroup$ – jbowman Jan 26 '18 at 4:28
  • $\begingroup$ To expand on that... if the constant is not $1/\sqrt{2\pi}\sigma_n$, what it really means is that $p(\mu|D)$ doesn't integrate to 1, which tells you that you have the wrong constant. The kernel / functional form is still that of a Gaussian distribution, though. $\endgroup$ – jbowman Jan 26 '18 at 4:36
  • $\begingroup$ @StackexchangeHouseNinja Did you read jbowman's comments, especially the first one? What do you think "$\propto$" means in $p(\mu \mid D) \propto p(\mu) p(D \mid \mu)$? $\endgroup$ – Juho Kokkala Jan 26 '18 at 20:09
  • 1
    $\begingroup$ I suspect I see your problem. It appears to me that you think $p(D|\mu)p(\mu)$ is a Normal distribution. It isn't. It's proportional to a Normal distribution. That is why we use the $\propto$ symbol instead of the $=$ symbol in the expression $p(\mu|D) \propto p(D|\mu)p(\mu)$. More specifically, $p(D|\mu)p(\mu) \propto N(\mu_n, \sigma^2_n) = p(\mu|D)$. $\endgroup$ – jbowman Jan 27 '18 at 2:25
2
$\begingroup$

So, just to note, $\Pr(\mu|D)$ is a real probability distribution or at least a distribution function. Read lines 8-12 in the commentary of

https://www.cs.ubc.ca/~murphyk/Papers/bayesGauss.pdf

which you posted and you should be able to resolve your questions.

$\endgroup$

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

  • $\begingroup$ However, he posted the solution his or herself. It was just overlooked. It was buried inside his own reference. $\endgroup$ – Dave Harris Jan 26 '18 at 20:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.