2
$\begingroup$

I've been working on a problem about coins and Bayes' theorem, but I'm getting some counterintuitive results. Here's the problem and what I've tried.

Imagine you have a coin, and you have no reasons to assume it's biased, so the prior is $P(heads) = 0.5$. Now imagine you've observed the data $D = \{heads:120, tails:100\}$, and you want to compute the expected heads probability after having observed this data. If we apply the Bayes rule we get

$$ P(head | D) = \frac{P(D|head) P(head)}{P(D)} $$

where

  • The prior is $P(head) = 0.5$, since we're assuming it's an unbiased coin. Therefore, $P(tail) = 0.5$ as well.
  • The likelihood is $P(D|head) = Binomial(220, 100, 0.5) = \binom{220}{120}0.5^{120}0.5^{100}$
  • $P(D) = P(D|head)P(head) + P(D|tail)P(tail)$, where $P(D|tail) = \binom{220}{100}0.5^{100}0.5^{120}$

If we plug everything in the Bayes rule we get

$$ P(head|D) =\frac{\binom{220}{120}0.5^{120}0.5^{100}0.5}{\binom{220}{120}0.5^{120}0.5^{100}0.5 + \binom{220}{100}0.5^{100}0.5^{120}0.5} = \frac{\binom{220}{120}}{\binom{220}{120} + \binom{220}{100}} = \frac{1}{2} $$

That seems counterintuitive to me. I would expect a posterior probability higher than $0.5$, something around $120/220\approx 0.545$.

Notice also that if we change the observed data to $D=\{heads: 12000, tails: 10000\}$ the results are the same even if we have much more information.

How is this possible? Is there some error in my numbers? Did I understand something wrong?

Thanks for your help!

Improve this question
$\endgroup$
8
  • 2
    $\begingroup$ Your prior asserts the coin is fair and offers no possibility of any other situation. A more useful prior would allow for various chances of heads. One standard such prior is the uniform (that is, Beta(1,1)) distribution on that chance. See our many posts on this topic for details. $\endgroup$
    – whuber ♦
    Jan 13 at 16:28
  • $\begingroup$ Thanks for your comment @whuber. My question is then why does this approach work in some cases, like the typical examples with clinical tests (thttps://davidbieber.com/snippets/2022-07-12-medical-test-paradox-paradox/), where the prior offer no possibility of any other situation, but the posterior is different than the prior. $\endgroup$
    – alexmolas
    Jan 14 at 16:45
  • 1
    $\begingroup$ I am unable to see the connection, because there is no such prior applied in the link you give. It is purely a probability calculation. $\endgroup$
    – whuber ♦
    Jan 14 at 18:37
  • 1
    $\begingroup$ No, it's not. That post does not concern estimating the frequency of disease in the population. $\endgroup$
    – whuber ♦
    Jan 14 at 18:52
  • 1
    $\begingroup$ The $1/10000$ is not a "prior:" it is given and not subject to estimation. $\endgroup$
    – whuber ♦
    Jan 14 at 21:23

1 Answer 1

Reset to default
1
$\begingroup$

You’ve given a prior distribution of a fair coin and no possibility of an unfair coin. No matter what kind of data you get, you can’t overwhelm the certainty in your prior.

Remember that you’re putting a prior on the probability parameter $p$ of some $\text{Binomial}(n,p)$ distribution, rather than speculating what the probability of a coin flip coming up heads will be. If you have a strong belief (but not certainty) that the coin is fair, you might want a prior that has considerable density on or near $0.5$. If your prior belief about the coin is, “Gee, I don’t know,” then you might want to consider all possible values in the parameter space to be equally likely, which translates to a prior distribution on $p$ of $\text{U}(0,1)$ (which equivalent to $\text{Beta}(1,1)$).

Improve this answer
$\endgroup$
6
  • $\begingroup$ ok! it makes sense, it's like if I had defined a delta prior at 0.5, so it's impossible to move it no matter how much data I observe. My question is now, why does this approach work in some cases, like the typical examples with clinical tests (towardsdatascience.com/…), where the prior is still a delta but Bayes theorem works properly. $\endgroup$
    – alexmolas
    Jan 13 at 17:25
  • $\begingroup$ Which part of the article do you mean? @alexmolas $\endgroup$
    – Dave
    Jan 13 at 17:30
  • $\begingroup$ when they talk about the medical paradox @Dave $\endgroup$
    – alexmolas
    Jan 13 at 17:51
  • $\begingroup$ @alexmolas Then you're calculating one probability of an event, rather than a posterior distribution of a parameter. (Maybe one could argue that this is a posterior distribution with all density on one value, but whatever...) $\endgroup$
    – Dave
    Jan 15 at 18:24
  • $\begingroup$ and why I can't use the same approach to compute the probability of a "head" event? This is what I'm missing. Thanks :) $\endgroup$
    – alexmolas
    Jan 16 at 7:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.