Going through some discussion on the classic dice roll or coin toss sequence. According to traditional probability theories, there is no connection between not rolling a 6 on the first dice roll, and getting a 6 on the next roll. The probability will be the same - 1/6. Each event is classed as being independent.

However, using coins as an easier example, when I look at Bayesian theory, if you had 99 coin tosses with heads arising, there would be some form of recalculation and the odds would change. It wouldn't be a 50-50 chance any more. I would like someone to explain exactly what the calculation would be and how you would come to it.

On a related issue, when I read about the Law of Large Numbers I see that there would be an 'expectation' that in my above example a tails toss would arise to return to the average. In many ways I can't see how you would disentangle this from falling into the Gambler's Fallacy whereby the gambler expects tails after so many heads in one long sequence.. The explanation seems to be that the Law of Large Numbers only works for large sets of trials but what exactly is a large set of trials? Would it be for 50 coin tosses? 5000 coin tosses? It seems a grey area to me.

All comments welcome!

  • $\begingroup$ Ask yourself: does the probability of something happening change depending on how you calculated it? $\endgroup$
    – Tim
    Commented Dec 25, 2016 at 21:44
  • $\begingroup$ People have trouble reconciling a long string of improbable outcomes with what will happen the next time. They think that nature has to change the probabilities so that things will even it. This is a fallacy that people sometimes call the law of averages. $\endgroup$ Commented Dec 26, 2016 at 1:51

2 Answers 2


Probability of throwing "6" on fair dice is $1/6$ and this has nothing to do with Bayes theorem. Bayes theorem is used to update our knowledge about conditional probabilities. If you are learning about properties of a die by observing it's outcomes, then you'll update your knowledge while observing new data, so the estimated probabilities will change. However the die itself does not change by observing it and calculating probabilities so the true probability does not change.

Your second question is answered in Does 10 heads in a row increase the chance of the next toss being a tail?


That isn't quite an accurate understanding. Bayesian methods can absolutely condition on $\Pr(heads)\equiv{\frac{1}{2}}$, although if that were your only parameter it would be a boring Bayesian question as you would have answered it by assumption.

I have used an example like this in a paper I wrote, so let me bring something like the paper over to here. For symbolic clarity, lets allow the bias of the coin toward heads to be symbolized by $\theta$ and a fair coin would have a bias for heads of $1/2$.

The traditional Frequentist or Likelihoodist solution would be to create a null hypothesis of $H_\emptyset:\theta=\frac{1}{2}.$ The Frequentist alternative would of course be $H_a:\theta\ne\frac{1}{2}$. Likelihoodists don't really have an explicit alternative, though it is pretty obvious that if you reject the null you are in the land of the Frequentist alternative since you have two mutually exclusive choices. The Frequentist and Likelihoodist can do this because the parameter space is not random, the sample space is random.

The Bayesian posterior probability that $\theta=\frac{1}{2}$ is zero if the parameter space is the closed set between zero and one. Any set of countable points, and there is only one point so you can obviously count the points, over a continuum of points has a measure of zero. So the Bayesian cannot test an exact point's probability. The Bayesian cannot directly test $\Pr(\theta)=\frac{1}{2}$. While a Bayesian could test the question of whether $\theta$ is in the region around $1/2$, you cannot test an exact point.

Let's change your question a little, however to put the Frequentist, Likelihoodist and the Bayesian all on the same footing. Let us imagine this coin is tossed in a casino. All the coins have been tested by engineers as perfectly balanced coins. Suspiciously though, the croupiers go by names such as Mandrake the Magician and Slick Eddy. You also hear that the casino is secretly a joint venture of the Japanese Yakuza and the Italian Mafia.

The coin is a fair coin, but you may not have a fair coin tosser. Both methodologies will assert similar hypothesis. The Frequentist will use the null hypothesis that the coin tosses are either fair or non-harmful to the customers, expressed as $H_\emptyset:\theta\le\frac{1}{2}$. The alternative, of course, is $H_a:\theta>\frac{1}{2}$. With 99 heads in a row you reject the null and if you have an estimator for $\theta$ of $\theta=1$.

The Bayesian question is not so simple. It would depend entirely upon what you believed before the first coin was tossed. Let us imagine that the idea of a con man or magician being a croupier had never crossed your mind, but you were aware, as the casino wanted you to be, that engineers had tested the coin and found it to be "fair." You have to ascribe prior beliefs to the coin and its tossers about what $\theta$ truly is over the parameter space. For simplicity, lets assume you ascribe a beta distribution prior distribution equivalent to fifty heads and fifty tails.

The formula for your prior is $2522283613639104833370312431400 (1 - \theta)^{49} \theta^{49}$. That is a lot of weight to overcome for the coin to be considered unfair. It grants a lot of certainty toward fairness if you plot that. Still, 99 heads in a row is enough to overcome that belief, with a posterior density of $12743385360030841036252908382886185223279736259800 (1 - \theta)^{49} \theta^{148} $ There is no real mass at or below 50%. But unlike your Frequentist cousin that now believes the coin will always come up heads, your expectation regarding the bias toward heads is $\frac{149}{199}$. You cannot shake your prior beliefs because they were sincere. It was your real prior.

If your prior beliefs are strong enough, it will take a lot of evidence to shake prior gullibility.

Now lets imagine you have a cousin who grew up in the woods and was home schooled and never heard of gambling or coin tossing. Your cousin ascribes an equal probability to each parameter value., or $\Pr(\theta=k)\propto{1},\forall{k}\in{[0,1]}$. That cousin would be very agnostic at the beginning and would be certain that the casino is up to no good at the end. That cousin's posterior distribution would be $100\theta^{99}$. His expected value for $\theta$ would be $\frac{100}{101}$ still allowing some chance for a slip up or a mistake, but excluding double headed coins as a possibility.

Their friend, on the other hand, was very worldly-wise and assigned a 99% prior probability that the place was going to cheat you. Their posterior and expectation are even more extreme. Just a disclosure, I didn't want to fire up Mathematica to do the numerical integration, but it would logically be more extreme. Sorry for the laziness.

Everyone is going to change their parameter estimates. How they change it will depend upon the rules they are operating under. What is very important here, though, is that it is still possible that 99 tosses were entirely by chance and that although improbable, very improbable even for the person with strong beliefs in fairness, that $\theta=\frac{1}{2}$.

If the coin were truly a fair coin with fair tossers, then the law of large numbers says that over time, that is as the number of tosses approaches infinity, that you would become certain from the data that its true value is $1/2$ regardless of your prior, assuming it were not a degenerate function, and regardless of "early" parameter estimates. To overcome 99 heads you would need something in the magnitude of $10^{30}$ random tosses.

  • $\begingroup$ So, in a nutshell...... The Gambler's Fallacy could be said to be true for 50 heads coin tosses with a false expectation that tails will come next. The Law of Large Numbers will kick in when the number of coin tosses approaches infinity....... This doesn't seem to make sense to me. The only difference between an apparently false view and an apparently true view being x amount of rolls? This precise number of rolls is also never explicitly defined? $\endgroup$
    – SimonK
    Commented Dec 30, 2016 at 21:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.