2
$\begingroup$

On these course notes, we are given the distribution of the posterior distribution https://people.eecs.berkeley.edu/~jordan/courses/260-spring10/lectures/lecture5.pdf

enter image description here

This famous result can be found in many other places, such as https://www.cs.ubc.ca/~murphyk/Papers/bayesGauss.pdf. However, I am confused about the coefficient (i.e., the $\dfrac{1}{\sqrt{2\pi}\sigma}$) term associated with the Gaussian for the posterior distribution. Here is what I am having trouble with.


We know that given $\mathcal{D} = (x_1, \ldots, x_n), x_i$ iid,

$$p(\mu|\mathcal{D}) \propto p(D|\mu)p(\mu)$$

Suppose that $$p(\mu) = \dfrac{1}{\sqrt{2\pi}\sigma_o} \exp{\dfrac{(x - \mu_o)^2}{2\sigma_o^2}}.$$ and $$p(D|\mu)= \dfrac{1}{(2\pi\sigma^2)^\frac{N}{2}} \exp(-\dfrac{1}{2\sigma^2}\sum\limits_{n = 1}^N (x_n - \mu))$$

then multiplying the expressions together, we obtain:

$$\dfrac{1}{(2\pi\sigma^2)^\frac{n}{2}} \exp{\left[-\dfrac{1}{2\sigma^2}\sum\limits_{n = 1}^N (x_n - \mu)\right]} \dfrac{1}{\sqrt{2\pi}\sigma_o} \exp{\left[\dfrac{(x - \mu_o)^2}{2\sigma_o^2}\right]}$$

While we can perform a complete the square inside of the exponential, what about the constants $\dfrac{1}{(2\pi\sigma^2)^\frac{n}{2}}$ and $\dfrac{1}{\sqrt{2\pi}\sigma_o}$?

I don't see how $\dfrac{1}{(2\pi\sigma^2)^\frac{n}{2}} *\dfrac{1}{\sqrt{2\pi}\sigma_o} = \dfrac{1}{\sqrt{2\pi}\sigma_n^2}$

where $\sigma_n^2 = (1/\sigma_o^2 + n/\sigma^2)^{-1}$ as shown in Lemma 6.

I tried playing around with the terms but I couldn't make them equal, even when $n = 1$. Have I made a mistake or is this because $p(\mu|\mathcal{D})$ is not a "true" probability distribution (i.e., doesn't integrate to 1)?

If so, how would people deal with this leading coefficient during simulation?

Addendum (see comment)

Bishop Pattern Reconigition and Machine Learning (2006) Pg. 98

enter image description here

$\endgroup$
15
  • 2
    $\begingroup$ It's just a constant of integration. Note that symbol "$\propto$" in your initial expression; it means "proportional to", as in "don't worry about the constant of integration until you're done with everything else, then it's whatever is needed to make the posterior integrate to one". $\endgroup$
    – jbowman
    Jan 26, 2018 at 4:25
  • $\begingroup$ It is $1/\sqrt{2\pi}\sigma_n$. $\endgroup$
    – jbowman
    Jan 26, 2018 at 4:28
  • $\begingroup$ To expand on that... if the constant is not $1/\sqrt{2\pi}\sigma_n$, what it really means is that $p(\mu|D)$ doesn't integrate to 1, which tells you that you have the wrong constant. The kernel / functional form is still that of a Gaussian distribution, though. $\endgroup$
    – jbowman
    Jan 26, 2018 at 4:36
  • $\begingroup$ @StackexchangeHouseNinja Did you read jbowman's comments, especially the first one? What do you think "$\propto$" means in $p(\mu \mid D) \propto p(\mu) p(D \mid \mu)$? $\endgroup$ Jan 26, 2018 at 20:09
  • 1
    $\begingroup$ I suspect I see your problem. It appears to me that you think $p(D|\mu)p(\mu)$ is a Normal distribution. It isn't. It's proportional to a Normal distribution. That is why we use the $\propto$ symbol instead of the $=$ symbol in the expression $p(\mu|D) \propto p(D|\mu)p(\mu)$. More specifically, $p(D|\mu)p(\mu) \propto N(\mu_n, \sigma^2_n) = p(\mu|D)$. $\endgroup$
    – jbowman
    Jan 27, 2018 at 2:25

1 Answer 1

2
$\begingroup$

So, just to note, $\Pr(\mu|D)$ is a real probability distribution or at least a distribution function. Read lines 8-12 in the commentary of

https://www.cs.ubc.ca/~murphyk/Papers/bayesGauss.pdf

which you posted and you should be able to resolve your questions.

$\endgroup$
1
  • $\begingroup$ However, he posted the solution his or herself. It was just overlooked. It was buried inside his own reference. $\endgroup$ Jan 26, 2018 at 20:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.