2
$\begingroup$

Assume we have a fair coin. We flip it 100 times. The outcome is all heads.

Why is it that all heads outcome is more surprising to us than a "more random looking" outcome with less regularity?

Aren't all outcomes of the same probability of $2^{-100}$?

And to make it a bit more statistical question, what intuition does tests like $\chi^2$ try to capture? If a sequence with a lot of regularity and a sequence with much less regularity both have the same probability, why would I distinguish between them? Why would I consider one more surprising than the other?

originally asked at MSE, but didn't get an answer that was satisfactory enough.

Improve this question
$\endgroup$
14
  • $\begingroup$ What do you mean by "regulatory"? Also, are you assuming that the coin flips are independent? If they are, then all 100 successive coin flips will have the same "surprise" according to their joint probability. Otherwise, if they are not independent, then not all 100 successive coin flips have the same "surprise" as measured by their joint probability. $\endgroup$
    – mhdadk
    Commented Mar 29, 2021 at 23:42
  • 4
    $\begingroup$ This website works best when users ask on question at a time. The introduction of the $\chi^2$ test is entirely unrelated to the question about coin flips. To help us answer the coin flip question, can you elaborate on what is unsatisfactory about the MSE explanation? $\endgroup$
    – Sycorax ♦
    Commented Mar 29, 2021 at 23:44
  • 1
    $\begingroup$ "Apophenia" is a psychological term for perceiving meaningful patterns where none exist. But questions about psychology are not on-topic here. If your question is statistical in nature, you'll need to edit your question to explain why you find it surprising for an event with probability $2^{-100}$ to occur once in $2^{100}$ experiments, as well as the exact setting of the experiment. $\endgroup$
    – Sycorax ♦
    Commented Mar 30, 2021 at 2:37
  • 2
    $\begingroup$ I think I can point to a fundamental reason that has mathematical and psychological appeal. Independent outcomes are exchangeable. Thus we are naturally led to perceive any sequence of outcomes $a_1a_2\ldots a_{100}$ as a representative of a class of outcomes--an "event"--comprising all permutations of that sequence $a_{\sigma(1)}a_{\sigma(2)}\ldots a_{\sigma(100)}.$ When the sequence includes $k$ heads, this event has cardinality $\binom{100}{k}$ and the chance of its associated event is $\binom{100}{k}2^{-100}.$ Although this doesn't fully address the psychology, it's a good start. $\endgroup$
    – whuber ♦
    Commented Mar 30, 2021 at 16:49
  • 1
    $\begingroup$ Yes, that's what I mean by "not fully address the psychology." The closest people have come to quantifying what "lacks regularity" might mean is embodied in random number testing: see en.wikipedia.org/wiki/Diehard_tests for instance. Another approach (explained for 2D arrays of numbers rather than 1D) is suggested at stats.stackexchange.com/questions/17109. $\endgroup$
    – whuber ♦
    Commented Apr 1, 2021 at 1:42

2 Answers 2

Reset to default
7
$\begingroup$

You're right; there's nothing special in terms of likelihood about 100 heads. You're right about that. One deserves to get equally excited about 20 heads, then 3 tails, then 6 more heads, then all tails.

This leads to my slant on this, based on Bayes' rule: seeing 100 heads in a row leads us to doubt our certainty that it's a fair coin with $f=0.5$—our hypothesis $\mathcal{H}$ about the data-generating process. There are other data-generating processes that could exist, which would better explain the 100 heads.

$$ p(\mathcal{H} \mid D) = \frac{p(D \mid \mathcal{H}) \times p(\mathcal{H})}{p(D)} $$

Even if all hypotheses are equally likely, it's more natural for these observations to come from a bent coin—or even a two-sided trick coin! That's why these surprise us. It makes us question our model of the world. With a more even dispersion of heads and tails (a member of the 'typical set' for this distribution), it does not lend the same credence to these alternative models.

The $\chi^2$ part of the question seems unrelated.

Improve this answer
$\endgroup$
4
$\begingroup$

Arya's answer is great. I'll offer the frequentist take. First off, all outcomes are not equiprobable under common assumptions. Some sequences are equivalent under the assumption that the former flip tells you nothing about the next flip. If this is a dubious assumption, then we could talk about probabilities of sequences, but under this assumption its the number of heads that matters most. We call this assumption "independence".

Under independence TTHHH is the same as HTHHT. The probability of seeing a given number of heads in a sequence is a well studied distribution called the binomial. I will leave that with you to research.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you. I have a BS in pure math, I know what iid and binomial distribution are. :) I know the number of heads distribution. I am comparing all heads to a single outcome, say 111001011011011000110... $\endgroup$
    – Kaveh
    Commented Mar 30, 2021 at 10:40

Not the answer you're looking for? Browse other questions tagged or ask your own question.