# Sample size to tell if more than X% of the population can do <thing>

I want to perform a test to determine (with 95% confidence) whether at least 70% of a population can perform some task.

The test involves sitting a randomly chosen person down and them attempting a task, which they either pass or fail. Equivalently, I flip a weighted coin.

The tests are expensive, so we will only perform a dozen or so of them.

The problem is to calculate pairs of (sample size, number of passes) that would give the required significance.

I want to be able to say something of the form

"Ask 10 people. If 8 or more pass then you can say with 95 confidence that the true ratio in the population is greater than 70%"

-
Do you want to take sampling schemes into account, or do you just want to understand the basic idea of a confidence interval for a binomial distribution? If you want to take real people and do this, and you want to be able to generalize to the population of your country or the world, you will have to work with much more sophisticated analyses. –  gung Jul 5 '12 at 16:48
Yes, it was just about the confidence interval for a binomial distribution, I didn't know that was the correct terminology. –  Phil Jul 5 '12 at 17:59

If your subjects are representative of the population and independent of each other then this is just the binomial test. You can get the information from you statement by just doing the tests with different inputs, here is an example using R:

> binom.test(8,10, alt='g', p=0.7)

Exact binomial test

data:  8 and 10
number of successes = 8, number of trials = 10, p-value = 0.3828
alternative hypothesis: true probability of success is greater than 0.7
95 percent confidence interval:
0.4930987 1.0000000
sample estimates:
probability of success
0.8


So 8 out of 10 only gives a lower limit of 49% in the 95% confidence interval.

However, you need to expand beyond just these simple statements. With 10 subjects you would need 10 out of 10 successes to say (with 95% confidence) that at least 70% can perform the task. But how likely are you to see 10 out of 10? that depends on the true proportion and you should really do a "what if" analysis with different "true" proportions to see how likely you are to see success.

If you are that limited on how many tests you can do then you may want to look to see if there is any other information available as an estimate that could be used as a prior in a Bayesian analysis.

-