I'm trying to follow the calculation of a Bayesian posterior for an intercept term which is updated by a single observation using the example from the following slide deck.

enter image description here

In this problem, the slope is fixed at 1. The intercept $b$ is a random variable.

The posterior distribution for $b$ is obtained from a normal prior distribution and a single data point (3,2). I'm having a hard time figuring out how to actually calculate those posterior values of -0.5 and 0.5 for the posterior mean and standard deviation. I'm not sure how Bayes rule is used to calculate these parameters.

Here are my attempts so far:

  • P(b|(3,2)) = P(b)*P((3,2)|b) / P((3,2))
  • I believe the (3,2) is equivalent to b=-1 because that's the only way the line can pass through (3,2) since the slope is fixed to 1. So the equation transforms to:
  • P(b|-1) = P(b)*P(-1|b) / P(-1)
  • Now it looks like we have a simpler problem: basically one random variable b, with a prior N(0,1) and a single data point, -1.
  • P(b) is the prior, so it is equal to N(0,1)
  • P(-1|b) is the probability of getting a -1 from a N(0,1) distribution. I'm a bit confused about this because I thought that the probability of getting a single value from a continuous distribution is always 0. So I'm not sure how to calculate this.
  • P(-1) --> Not sure where to start with this. I can't explain it conceptually let alone mathematically.
  • $\begingroup$ (-1) It seems hard to believe that you want to understand the concept of a posterior distribution from a single slide. Have you considered opening a standard textbook? $\endgroup$ – Xi'an Apr 16 '18 at 6:04
  • 2
    $\begingroup$ @Xi'an, I'm not sure why you down-voted... I've explained my question clearly and provided my thought process. I never said I'm trying to understand the concept of a posterior distribution from a single slide, so I don't know where you got that idea from. I have already learned about priors/posteriors on a conceptual level. I'm now trying to take that conceptual understanding and apply it to a real example that I found in that slide. $\endgroup$ – DataMan Apr 16 '18 at 10:39
  • 1
    $\begingroup$ The update for a Normal distribution is well-known and in probably every textbook on Bayesian statistics ever written... the fact that you don't seem to have run across it is where (I assume) @Xi'an 's puzzlement over your question comes from. Entering "Bayesian normal distribution" into Google gives a lot of hits, in my case the third one of which - ams.sunysb.edu/~zhu/ams570/Bayesian_Normal.pdf - seems applicable. $\endgroup$ – jbowman Apr 16 '18 at 18:15
  • $\begingroup$ Addendum: to call this example linear regression is a mistake as$$y-x\sim{\cal N}(b,1)$$ means this is a regular normal mean estimation problem. $\endgroup$ – Xi'an Apr 16 '18 at 18:21
  • 1
    $\begingroup$ It is not for the prior. Look at the equation! The error $e \sim N(0,1)$. Your prior also has a standard deviation of 1, but that's not the only thing that does. The link does not say we know the standard deviation for the posterior - we can't know that before we calculate the posterior, which is what the link shows you how to do. It says we know the standard deviation of the probability distribution of the data. $\endgroup$ – jbowman Apr 16 '18 at 18:58

You can derive the posterior distribution of $b$ in the following way: $$ p(b=b_1|(3,2)) \propto \frac{p((3,2)|b=b_1)p(b=b_1)}{p((3,2))} $$ Since $p((3,2))$ does not depend on $b$, then:
$$ p(b=b_1|(3,2)) \propto p((3,2)|b=b_1)P(b=b_1) $$

Also: $$ p((3,2)|b=b_1) = p(y=2|x=3, b=b1) $$

From $y = x + b +\epsilon$, and given $x = 3, b = b_1, \epsilon \sim N(0, 1)$, one can infer that $y \sim N(3+b_1, 1)$. Therefore, $$ p(b=b_1|(3,2)) \propto e^{-\frac{(2-3-b_1)^2}{2}} \times e^{-\frac{(b_1)^2}{2}} $$ $$ \propto e^{-\frac{(b_1 - (-0.5))^2}{2 \times 0.5}}$$

Using the above, you can infer that $b \sim N(-0.5, 0.5)$.

  • $\begingroup$ Thanks @Ankit. I'm unsure about two things, it would be great if you could clarify them. First, what does b1 stand for? Second, how did you get "given x=1". Thanks again! $\endgroup$ – DataMan Apr 16 '18 at 3:44
  • $\begingroup$ You cannot write $P(b=b_1)$ in a continuous setup. Use densities instead. And observing $(y,x)=(2,3)$ is equivalent to observing a Normal ${\cal N}(b,1)$ realisation $y-x$ equal to $2-3=-1$, hence using the Normal likelihood at $-1$ is correct. $\endgroup$ – Xi'an Apr 16 '18 at 6:06
  • $\begingroup$ Thanks, @Xia'an for pointing it out. I missed the point that these are continuous distributions. I have made the edit. $\endgroup$ – Ankit Goyal Apr 16 '18 at 17:58
  • $\begingroup$ @DataMan, you can think of b1 as any real value. Essentially we are calculating the probability density at some fixed real value and then using it to infer the complete distribution. x=1 was a typo and I have edited it. $\endgroup$ – Ankit Goyal Apr 16 '18 at 17:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.