8
$\begingroup$

The rule of three in statistics states that if an event is binomially distributed and does not occur with in $n$ trials the maximum chance of it occurring is approximately $3/n$. Suppose we have a roulette table with only two options, red or black. The chance of either of these occurring is clearly 1/2. Suppose, however, that we don't see black for 10 turns of the wheel. We might then decide to reason, ignoring the prior knowledge of the distribution, that the chance of black occurring is at most 3/10, which is not true. Is this a misapplication of the rule? If so, why, and how does one determine when it is proper to apply it.

$\endgroup$
  • 2
    $\begingroup$ If you know the probabilities apriori, why would you make that argument? $\endgroup$ – rightskewed Mar 19 '15 at 15:53
  • $\begingroup$ @rightskewed You wouldn't reasonably, but it came from my misunderstanding that the rule meant you would never find values greater than 3/n (leading me to think it was being violated) when in fact you would see such values 5% of the time. $\endgroup$ – 114 Mar 19 '15 at 17:44
15
$\begingroup$
The rule of three in statistics states that if an event is binomially  
distributed and does not occur with in n trials the maximum chance of it     
occurring is approximately 3/n. 

No, that's not what it says. It says that a 95% confidence interval for the actual chance of it occurring is approximately [0, 3/n]. That is not the same thing. The largest value for the 'chance of occurring' contained in the interval is indeed 3/n, though the question of which of the values within the interval is most likely is not answered.

The rule says: 'guess that the true chance of occurring is 3/n or less and you will be wrong about 5% of the time.

Suppose we have a roulette table with only two options, red or black. The 
chance of either of these occurring is clearly 1/2. 

Exactly, so there is no need for a confidence interval because the 'chance of occurring' is known. You could, on the other hand, test the coverage of the approximate interval that the rule provides using such a wheel.

Suppose, however, that we don't see black for 10 turns of the wheel. 
We would then reason that the chance of black occurring is at most 3/10,     
which is not true. Is this a misapplication of the rule? If so, why, and 
how does one determine when it is proper to apply it.

It is a misapplication of the idea of a confidence interval, which is applied to bound the range of plausible values of things that are unknown, and which in any particular application need not contain the true value if it becomes known.

$\endgroup$
  • $\begingroup$ That makes sense, so in fact if we didn't know the distribution of the roulette wheel and suspected it was rigged in some way we might want to use the rule of three in order to get the 'best guess'. $\endgroup$ – 114 Mar 19 '15 at 17:43
  • $\begingroup$ @114 You don't use confidence intervals to come up with a "best guess". In the unlikely case you truly have no prior beliefs whatsoever, then a single "best guess" for the rate of black after observing none for 10 trials would be somewhere between 0 and 5% (depending on your philosophy and what you mean by "best", e.g. mode-median-mean of posterior), whereas the upper end of the 95% confidence interval might might somewhere between 20-30% (depending on how you estimate it -- rule of 3 is probably not a good choice for such small n). $\endgroup$ – A. Webb Mar 19 '15 at 19:14
  • $\begingroup$ @114 However, you should probably have very strong prior belief that actual rate is 50%, whereupon the 95% confidence interval after observing a run of 10 would still contain 50% and be tight around it. $\endgroup$ – A. Webb Mar 19 '15 at 19:14
  • $\begingroup$ @A.Webb Could you explain your last comment a bit further? $\endgroup$ – 114 Jun 25 '15 at 1:38
  • $\begingroup$ Your restatement of the rule is incorrect. To see why, suppose you are an expert who is consulted only when the results are unexpected. Thus, whenever you are presented with a dataset showing zero occurrences of something, it is in a circumstance where that surprised somebody. Because of this selection process it might happen that in most cases where you invoke the rule the true chance actually exceeds $3/n.$ Then in most of your applications--not just 5% of them--you will be wrong. $\endgroup$ – whuber Dec 1 '19 at 17:04
0
$\begingroup$

From Wikipedia, an example of "the rule of 3" is described as "For example, a pain-relief drug is tested on 1500 human subjects, and no adverse event is recorded. From the rule of three, it can be concluded with 95% confidence that fewer than 1 person in 500 (or 3/1500) will experience an adverse event."

The following derivation helped me understand the "rule of three" from another perspective.

Assume n = 1500, and the worst possible case is that an adverse event is found when n = 1501, i.e., p_ = 1/1501 (=0.000666).

The Standard Error (SE) of p_ can be calculated as the square root of p_*(1-p_)/n (=0.000666), which can be approximated as 1/1500 (= 0.000667), that is, SE ~ 1/n.

Assuming that p_ (the observed value of the true p) has a normal distribution centered at p, then we will have a 97.8 % confidence (one-sided upper bound of 2 SE) that, if we repeat the sampling, the true p will be smaller than 1/n + 2*SE = 1/n + 2/n = 3/n (= 0.002).

If the p_ value (=1/1501) is used to calculate the value of the upper bound, the result is 0.001998, which is very close to 0.002.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.