8
$\begingroup$

I'm still trying to understand prior and posterior distributions in Bayesian inference.

In this question, one flips a coin. Priors:

unfair is 0.1, and being fair is 0.9

Coin is flipped 10x and is all heads. Based on this evidence, P(unfair | evidence) ~ 0.653 and P(fair | evidence) ~ 1-0.653.

Apparently:

Both priors (originally 0.1 and 0.9) must be updated. – user3697176 Apr 27 at 4:38

  1. What exactly does it mean to update a prior? I'm not sure I understand Wiki. Like we replace 0.1 and 0.9 with 0.653 and 1-0.653 and then run another experiment and continue to do so until the priors stabilize?

Question 2 might just follow from 1 but anyway:

  1. Why must they be updated? I'm guessing because 0.1 and 0.9 are sufficiently far from 0.653 and 1-0.653.

If that is right, how far must they be from 0.1 and 0.9 to say that priors must be updated? What if evidence gave us 0.11 and 0.89?

If that is wrong, why then?

$\endgroup$
6
  • 2
    $\begingroup$ "update the prior" means "incorporate the information we have from the data to turn our prior belief into a posterior belief". If you initially believe there's a 10% chance of rain where you are, but then see large banks of dark clouds rolling in, you use that update your prior belief ($P(\text{rain})=0.1$) to a posterior belief ($P(\text{rain}|\text{dark clouds})=0.6$,say). Bayesian statistics does the same kind of thing except that the part with the incorporation of the data is done more formally (using a model). $\endgroup$
    – Glen_b
    Aug 9, 2015 at 6:02
  • $\begingroup$ Thanks @Glen_b. So 'Both priors (originally 0.1 and 0.9) must be updated. – user3697176 Apr 27 at 4:38' is wrong? :| $\endgroup$
    – BCLC
    Aug 10, 2015 at 4:13
  • 2
    $\begingroup$ That's actually one prior distribution across two possible outcomes whose probabilities must add to one. So saying "both priors" is strictly incorrect (there aren't two of them), but if we allow for some vagueness of expression, the likely intent seems correct -- updating one correctly gives you the other as well. $\endgroup$
    – Glen_b
    Aug 10, 2015 at 5:23
  • 3
    $\begingroup$ The point of the Bayesian analysis is to update the prior with the information in the data. $\endgroup$
    – Glen_b
    Aug 11, 2015 at 11:10
  • 1
    $\begingroup$ BCLC, @Glen_b 's last comment is right at the heart of your question. I encourage you to consider what not updating one's priors means in plain language: when I already believe something, no amount of evidence can convince me to change my mind. $\endgroup$
    – Alexis
    Dec 12, 2022 at 23:16

2 Answers 2

19
$\begingroup$

In plain english, update a prior in bayesian inference means that you start with some guesses about the probability of an event occuring (prior probability), then you observe what happens (likelihood), and depending on what happened you update your initial guess. Once updated, your prior probability is called posterior probability.

Of course, now you can:

  • stop with your posterior probability;
  • use you posterior probability as a new prior, and update such a probability to obtain a new posterior by observing more evidence (i.e. data).

Essentially, updating a prior means that you start with a (informed) guess and you use evidence to update your initial guess. Recall that

$$ p(\theta | x) = \frac{p(x|\theta)p(\theta)}{p(x)},$$

where $p(\theta)$ is your prior, $p(x|\theta)$ is the likelihood (i.e. the evidence that you use to update the prior), and $p(\theta|x)$ is the posterior probability. Notice that the posterior probability is a probability given the evidence.

Example of coins: You start with the guess the probability of the coin being fair is $p = 0.1$. Then, you toss 10 times the coin, and you obtain a posterior probability $p = 0.3$. At this point, you can decide to be satisfied with $p = 0.3$ or toss the coin again (say 90 times): in this case, your prior will be $p = 0.3$ -- i.e. the posterior becomes the new prior -- and you will obtain a new posterior probability depending on new evidence.

Suppose that after 1000 tosses your posterior probability is $p = 0.9$. At the beginning you prior was $p = 0.1$, so you were supposing that the coin was unfair. Now, based on the evidence of 1000 tosses, you see that the probability of the coin being fair is high.

Notice that the fact that you can easily update a probability as you have new evidences is a strength of the bayesian framework. The point here is not that the prior must be updated, but to use all the available evidence to update your guess about a certain probability.

$\endgroup$
4
  • $\begingroup$ So 'Both priors (originally 0.1 and 0.9) must be updated. – user3697176 Apr 27 at 4:38' is wrong? :| Thanks stochazesthai. $\endgroup$
    – BCLC
    Aug 10, 2015 at 4:12
  • 1
    $\begingroup$ The point there is that if the probability (prior) of the coin being fair is $p = 0.1$, then the probability (prior) of the coin being unfair is $1 - p = 1 - 0.1 = 0.9$. Once you have updated $p$, the probability of the coin being fair, you have also updated the probability of the coin being unfair. :) $\endgroup$ Aug 10, 2015 at 4:44
  • $\begingroup$ This was exceptionally explained. Interestingly, the phrase update priors is used frequently in the rationalist community [1]. I wasn't expecting to find a math definition. Nevertheless, understand it. However, this explanation was brilliant. Mainly because it holds even without needing to understand the math part. [1] youtube.com/watch?v=BrK7X_XlGB8 -> interesting video that looks at updating priors in a non-math sense $\endgroup$
    – Tomiwa
    Jan 6, 2023 at 22:22
  • $\begingroup$ *correction, I rewatched the video and it does explain it in a math sense (so my earlier thinng about "updating priors in a non-math sense" was not 100% true). I was recommending it based on memory/context before watching. However, it does a great job of tying it to real world examples. E.g. using "bayesian thinking" to make better life decisions. It's by Julia Galef, in case anyone in the future comes across this thread and finds the idea of Bayesian thinking. It's one of the most important mental models i've learned in my life and I try to raise awareness about it. $\endgroup$
    – Tomiwa
    Jan 6, 2023 at 22:36
1
$\begingroup$

Because I think we want a model that incorporated the data observed so that the model (probability distribution) fits the data observed and we can use the model for stable predictions, as the initial prior is just a hypothesis to start with.

The steps to update the prior distribution to get the posterior.

enter image description here

$\endgroup$
1
  • $\begingroup$ +1 For sharing these beautiful notes. $\endgroup$
    – Alexis
    Dec 26, 2022 at 18:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.