4
$\begingroup$

I'm still trying to understand prior and posterior distributions in Bayesian inference.

In this question, one flips a coin. Priors:

unfair is 0.1, and being fair is 0.9

Coin is flipped 10x and is all heads. Based on this evidence, P(unfair | evidence) ~ 0.653 and P(fair | evidence) ~ 1-0.653.

Apparently:

Both priors (originally 0.1 and 0.9) must be updated. – user3697176 Apr 27 at 4:38

  1. What exactly does it mean to update a prior? I'm not sure I understand Wiki. Like we replace 0.1 and 0.9 with 0.653 and 1-0.653 and then run another experiment and continue to do so until the priors stabilize?

Question 2 might just follow from 1 but anyway:

  1. Why must they be updated? I'm guessing because 0.1 and 0.9 are sufficiently far from 0.653 and 1-0.653.

If that is right, how far must they be from 0.1 and 0.9 to say that priors must be updated? What if evidence gave us 0.11 and 0.89?

If that is wrong, why then?

$\endgroup$
  • 1
    $\begingroup$ "update the prior" means "incorporate the information we have from the data to turn our prior belief into a posterior belief". If you initially believe there's a 10% chance of rain where you are, but then see large banks of dark clouds rolling in, you use that update your prior belief ($P(\text{rain})=0.1$) to a posterior belief ($P(\text{rain}|\text{dark clouds})=0.6$,say). Bayesian statistics does the same kind of thing except that the part with the incorporation of the data is done more formally (using a model). $\endgroup$ – Glen_b -Reinstate Monica Aug 9 '15 at 6:02
  • $\begingroup$ Thanks @Glen_b. So 'Both priors (originally 0.1 and 0.9) must be updated. – user3697176 Apr 27 at 4:38' is wrong? :| $\endgroup$ – BCLC Aug 10 '15 at 4:13
  • 1
    $\begingroup$ That's actually one prior distribution across two possible outcomes whose probabilities must add to one. So saying "both priors" is strictly incorrect (there aren't two of them), but if we allow for some vagueness of expression, the likely intent seems correct -- updating one correctly gives you the other as well. $\endgroup$ – Glen_b -Reinstate Monica Aug 10 '15 at 5:23
  • $\begingroup$ @Glen_b Sorry for the confusion. I mean, user3697176 seems to be saying that both priors must be updated because they are far from initial guess. Why must they be updated? Just because they are unequal from 0.1 and 0.9? Because they are far from 0.1 or 0.9? I understand I may sound like somewhat brainwashed by frequentist thinking, but must the priors be updated if the evidence gives us 0.11 and 0.89 (numbers close, by some standard, to 0.1 and 0.9) $\endgroup$ – BCLC Aug 11 '15 at 11:05
  • 2
    $\begingroup$ The point of the Bayesian analysis is to update the prior with the information in the data. $\endgroup$ – Glen_b -Reinstate Monica Aug 11 '15 at 11:10
11
$\begingroup$

In plain english, update a prior in bayesian inference means that you start with some guesses about the probability of an event occuring (prior probability), then you observe what happens (likelihood), and depending on what happened you update your initial guess. Once updated, your prior probability is called posterior probability.

Of course, now you can:

  • stop with your posterior probability;
  • use you posterior probability as a new prior, and update such a probability to obtain a new posterior by observing more evidence (i.e. data).

Essentially, updating a prior means that you start with a (informed) guess and you use evidence to update your initial guess. Recall that

$$ p(\theta | x) = \frac{p(x|\theta)p(\theta)}{p(x)},$$

where $p(\theta)$ is your prior, $p(x|\theta)$ is the likelihood (i.e. the evidence that you use to update the prior), and $p(\theta|x)$ is the posterior probability. Notice that the posterior probability is a probability given the evidence.

Example of coins: You start with the guess the probability of the coin being fair is $p = 0.1$. Then, you toss 10 times the coin, and you obtain a posterior probability $p = 0.3$. At this point, you can decide to be satisfied with $p = 0.3$ or toss the coin again (say 90 times): in this case, your prior will be $p = 0.3$ -- i.e. the posterior becomes the new prior -- and you will obtain a new posterior probability depending on new evidence.

Suppose that after 1000 tosses your posterior probability is $p = 0.9$. At the beginning you prior was $p = 0.1$, so you were supposing that the coin was unfair. Now, based on the evidence of 1000 tosses, you see that the probability of the coin being fair is high.

Notice that the fact that you can easily update a probability as you have new evidences is a strength of the bayesian framework. The point here is not that the prior must be updated, but to use all the available evidence to update your guess about a certain probability.

$\endgroup$
  • $\begingroup$ So 'Both priors (originally 0.1 and 0.9) must be updated. – user3697176 Apr 27 at 4:38' is wrong? :| Thanks stochazesthai. $\endgroup$ – BCLC Aug 10 '15 at 4:12
  • 1
    $\begingroup$ The point there is that if the probability (prior) of the coin being fair is $p = 0.1$, then the probability (prior) of the coin being unfair is $1 - p = 1 - 0.1 = 0.9$. Once you have updated $p$, the probability of the coin being fair, you have also updated the probability of the coin being unfair. :) $\endgroup$ – stochazesthai Aug 10 '15 at 4:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.