# How many observations needed to reach 90% confidence?

Let's say I have a pool of 1,000 marbles. Most likely they are all red. Maybe a few black. How many marbles do I need to look at (assuming each outcome is red) before I'm 90% confident that 99% of the marbles are red? What is this type of analysis called? Have I defined all the necessary parameters?

• My answer to quora.com/… doesn't answer your question directly, but should help. Since it's not part of SE, you may want to take it, modify it, and post it as an answer here. – barrycarter Jan 16 '16 at 21:20

The way you worded your question sounds a bit off. Taken literally, we never have a confidence that a parameter equals a point value. Usually, we talk about confidence intervals, or the probability that a parameter lies within a given range. As you know, the larger our sample, the smaller that internal will be (the more precise our estimate will be).

So the problem with your working is that we can draw large samples and get a confidence interval from, say [80%-100%], a larger one gives us [90%-100%]. A really large one gives [95%-99%] maybe a huge one [96%-98%] (all using 90% confidence). So only that last draw with a huge sample gives us a range that excludes your query of 99%. But that last example also suggests that you have more than 1% black marbles.

I wasn't sure whether you meant to ask "90% confident that at least 99% of the marbles are red"?

To pursue your analysis (which is known as a statistical power analysis), we do need to make an assumption about the true proportion of black marbles. If we assume it's zero, then the power analysis will simply conclude all samples of any size are 100% red. That does us no good.

So we pick a different starting assumption: 1% black. Any sample we draw has an expected proportion of 99% red, but we have to construct a sampling confidence interval around that that depends on the degree of confidence (90%) and the true proportion of red marbles we're assuming (99%). Again, we use the "true" or asserted proportion because we're asking a hypothetical question.

The usual Normal approximation to the confidence interval would be .99 +/- z(.95)*sqrt(p(1-p)/n), where z(.95) is the inverse normal CDF that leaves 5% in each tail = 1.64. But this leaves you with a confidence interval that will likely exceed 1.0 on one side and that ought to tell you something's fishy.

Turns out the normal approximation isn't great when p is very small or large (near 0 or 1); some people use the Poisson distribution in this case, and there are other approximations to the confidence interval outlined in this Wikipedia article: https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval

My experience is that the Poisson is a better approximation for these "rare event" scenarios. Fix the sample size at, say, 100. If we're assuming 1% of balls are black, then that's 1 ball in our sample. A Poisson(1) distribution has a 37% chance of seeing 0 blacks, 37% seeing 1 black, and 18% seeing 2 blacks. The cumulative probability of those 3 cases is 92%. So if we observe 3 or more balls being black in a sample of 100, we could reject our starting assumption that k=1 with 92% confidence.

The probability of 3 blacks in that distribution is 6%, and the CDF up to 3 blacks is 98%, so if we observe 4 or more balls we can reject the hypothesis of 1% black with 98% confidence.

Bottom line - this all depends on my interpretation of your question being correct, and there are multiple approximations for these "rare event" scenarios. (see the Wikipedia article)

• "99% of marbles are red" is usually understood to mean "at least 99% of marbles are red." An important part of the question's phrasing, "look at ... before," suggests it is asking for a sequential sampling approach, not a power analysis (which could be carried out accurately with a Binomial model in any event). – whuber Jan 16 '16 at 22:29
• Thank you for the correction - I see now the poster meant "continue to sequentially draw red marbles". – Larry Jan 18 '16 at 7:44

You would have to pick 188 red marbles in a row to be 90% confident that at least 99% (or 990) of the 1000 marbles were red.

This is a Bayesian approach, and is effectively a copy of my answer to https://www.quora.com/I-have-burned-200-disks-and-I-want-to-make-sure-that-they-are-all-in-perfect-working-order-What-is-the-smallest-size-sample-I-could-test-in-order-to-be-relatively-confident-that-98-of-all-the-disks-are-fine-burned-correctly

Suppose you randomly pick 50 marbles and they are all red.

If there were exactly 100 red marbles (900 black), you could do this in Binomial[100,50] or 100891344545564193334812497256 ways.

If there were exactly 101 red marbles (899 black), you could do this in Binomial[101,50] or 199804427433372226016001220056 ways.

In other words, it's 199804427433372226016001220056/100891344545564193334812497256 (which is 101/51) more likely there are exactly 101 red marbles than there are exactly 100 red marbles.

In general, there are Binomial[n,50] ways there can be exactly n red marbles if you picked 50 marbles and all were red.

How many ways can there be at least 990 red marbles (990 = 99% of 1000):

Sum[Binomial[i,50],{i,990,1000}] = 81531993898242794598076623570369772515365019903144264043266969772686950795976808446840

How many ways can there be n red marbles for any value of n?

Sum[Binomial[i,50],{i,0,1000}] = 185684734874564765959015411764819077718596543052359268997222801777737175488143399579280

(which also happens to be Binomial[1001,51])

So the chance we have 990 or more red marbles (if we pick 50 red in a row) is 81531993898242794598076623570369772515365019903144264043266969772686950795976808446840/185684734874564765959015411764819077718596543052359268997222801777737175488143399579280 or about 43.91%

Now, if we change 50 marbles to k marbles, the chance of 990 or more red marbles is:

Sum[Binomial[i,k],{i,990,1000}]/Binomial[1001,k+1]

Some results:

• To be 50% confident you have 990+ red marbles, pick 60 red marbles.
• To be 75% confident, pick 117 red marbles.
• To be 90% confident, pick 188 red marbles. (this is the answer to your question).
• To be 95% confident, pick 237 red marbles.
• To be 99% confident, pick 340 red marbles.
• To be 99.5% confident, pick 380 red marbles.
• To be 99.9% confident, pick 464 red marbles.
• To be 99.95% confident, pick 496 red marbles.
• To be 99.99% confident, pick 564 red marbles.
• To be 99.995% confident, pick 591 red marbles.
• To be 99.999% confident, pick 646 red marbles.

And, of course, to be 100% confident that 99% of the marbles are red, pick 990 red marbles.

Given how big these numbers get, using a normal distribution approach may have been better, though slightly less accurate.