If you flip a fair coin indefinitely, how likely is it that, at some point, you will have evidence that the coin is not fair?

What I mean is this: Take a fair coin and select a $p$-value. If we were to flip the coin $100$ times, there would be some threshold proportion $Q$ such that if the propotion of heads (or tails) were larger than $Q$, we would have evidence (given our selected $p$-value) that the coin was not fair. If we were to flip the coin $1000$ times, the threshold proportion would diminish. As the number of flips $N$ increases without bound, the $Q$ approaches $.5$.

If we flip the fair coin indefinitely, what is the probability that there will be some $N$, such that, if we had stopped flipping at $N$, we would have ended up with evidence against the coin's being fair (in terms of a $p$-value)?

  • 3
    $\begingroup$ If $N$ is selected in advance, & you hold to that decision w/o regard for what the outcome looks like at that moment, the probability of a type I error will be $\alpha$. OTOH, if you continue until $p<\alpha$, & stop exactly when it becomes such, w/o regard for what the $N$ is at that moment (& are allowed / willing to continue indefinitely), the probability of a type I error is $1$. $\endgroup$ – gung Jul 16 '13 at 16:38
  • 1
    $\begingroup$ A closely related recent question on math.SE. $\endgroup$ – cardinal Jul 16 '13 at 16:51
  • 2
    $\begingroup$ Some of the ideas here could be fixed more precisely to make the question a little more answerable. For example, it is possible to choose a sequence of constants $q_n$ such that the probability of the random walk $S_n$ ever exceeding $q_n$ is bounded by a fixed, given $\alpha \in (0,1)$. Of course, choosing $q_n = n$ is a trivial such sequence, but the sequence need not be so trivial. (This is just a consequence of the union bound in action.) On the other hand, you are likely thinking of fixing $q_n$ to give a Type I error rate of $\alpha$ for each $n$, but such a $q_n$ doesn't exist exactly. $\endgroup$ – cardinal Jul 16 '13 at 16:55
  • $\begingroup$ cardinal, thank you for your comment and for directing me to that question on math.SE. His question is basically what I was trying to ask. $\endgroup$ – Zach Barnett Jul 16 '13 at 18:50
  • 1
    $\begingroup$ This appears to be related to sequential testing [1] [2] $\endgroup$ – Glen_b Oct 6 '13 at 22:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.