# Sample size when one hypothesis is overwhelming

I am looking at a situation where I have a population of about 150-170. I am testing for the absence or presence of a particular property. Given the population, I can easily figure out the sample size needed for various confidence levels.

However, I have read in a number of places (e.g., wikipedia) that that a higher signal to noise ratio can reduce the sample size needed. This makes sense intuitively, but I can't find information on how to apply it aside from very general formulas ($c = \frac{s}{n} * \sqrt{\text{sample size}}$). For example, if I take a sample size of 20, and all 20 samples have the property, how does this impact the confidence? How can I figure out the sample size given an assumption about the probably of the property being true (e.g., I assume that there is a 100% chance that it is true)?

I have a hunch that this is a related to Finite Sample Size in Hypothesis Testing, but I am not sure.

Both practical and theoretical explanations are welcome. However, I minored in statistics, but about 15 years ago.

-

If you assume the property is true of 100% of the population, then why sample?

What you can do is consider confidence. This reframes your question:

Given that I have observed a representative sample of $20$ subjects and none of them had the property, just how large a proportion of the population can have that property and still be consistent with what I observed?

The experiment can be modeled as independent random draws from a box containing one ticket for each member of the population. The tickets for members with the property have $1$ written on them and the tickets for members without the property have $0$ written on them. Your observed count is the total on the tickets that were drawn.

When the population is large compared to the sample (or the sample is performed with replacement), the chance of a zero count is particularly easy to compute: letting $p$ be the proportion of members of the population with the property, the chance that any one ticket exhibits that property is $p$. Because independent probabilities multiply, the chance that all tickets out of $n$ drawn exhibit the property therefore equals $p^n$. If this chance is small, your sample is not very consistent with the value of $p$.

For example, suppose you decide that an $\alpha$ chance, such as $\alpha=0.01$ (or whatever small number you like) is small enough to demonstrate inconsistency. That implies $p^n \ge \alpha$ for any consistent value of $p$, with solution

$$p \ge \alpha^\frac{1}{n}.$$

Taking $n=20$ and $\alpha=0.01$, for instance, gives

$$p \ge 0.01^\frac{1}{20} \approx 0.79.$$

In other words, if less than $79$% of the population has the property, then it's very unlikely (less than a $1$% chance) that every member in a sample of $20$ would have that property. Another way to say the same thing is that when you observe that every member of a sample of $20$ has the property, then it's fair to conclude that at least $79$% of the population has that property.

The confidence is the complement of $\alpha$, equal to $1-\alpha$. The conventional way to state the preceding conclusion is

With $1 - \alpha = 99$% confidence, at least $79$% of the population has the property.

The same reasoning allows us to estimate a minimum sample size $n$ provided we (a) specify the confidence we want to have in the conclusions and (b) stipulate what value of $p$ we wish to detect. You must specify both of these numbers to determine a minimum sample size.

For example, suppose we hope to conclude, with $95$% confidence, that at least $99$% of the population has the property. Here $\alpha = 1-95/100 = .05$ and $p = 99/100 = 0.99$. Solving for $n$ gives

$$n \ge \frac{\log \alpha}{\log p}.$$

For these values of $p$ and $\alpha$, $n \ge 298.07$, implying $n \ge 299$ (because $n$ is integral, of course).

This calculation illustrates the rule of three: the value of $299$ is close to three times $1/(1-p) = 100$. The number $1/(1-p)$ is the expected number of subjects we would have to inspect before finding a single one without the property. (E.g., we would expect to examine $100$ subjects to find a characteristic that occurs only once in every $100$ people in the population). This rule applies to the common demand for $95$% confidence: to achieve that level of confidence, you need to examine three times the expected number of subjects. That's the rule of three.

Technical note: the rule of three follows from the fact that $\log .05 \approx -3$ and $\log p \approx p-1$ when $p$ is close to $1$, whence

$$n \ge \frac{\log \alpha}{\log p} \approx \frac{-3}{p-1} = \frac{3}{1-p}.$$

If you prefer other levels of confidence (equal to $1-\alpha$), you can develop your own simple rule of thumb by computing $-\log \alpha$. E.g., if you prefer $99$% confidence levels, then your $\alpha$ equals $0.01$ with a logarithm of $-4.6$, giving a "rule of $4.6$." The rest you can now do in your head. For instance, the minimum sample size to conclude that at least $p = 90$% of a population has a property is $4.6 / (1 - 0.90) = 46$. You can now use simple mental math to evaluate the tradeoffs among your assumed value of $p$, your desired confidence $\alpha$, and the costs of sampling $n$ subjects.

Important comment

You cannot conclude $100$% of a population has a property unless you take a complete census. If you fail to sample even one individual, that individual might fail to have the property.

-
Thanks for the excellent answer! I am going to run though what I did to make sure that I understand - I will try and post tomorrow morning. Thanks! –  Erick T Aug 12 '12 at 20:01