I have a population of say 100,000 records that can be either Positive or Negative (true probability of Positive is very small, but unknown - may even be zero).

I want to take a sample and estimate how many Positive cases are in the population, to be able to say something along the lines of "I'm 95% confident there are less than 50 positive cases in the population" or even better show my confidence there are no positive cases eg. "I'm 87% confident there are no positive cases".

How can I do the following:

  • What tests do I want to be doing here? From doing some research it looks like I want to use a Proportion Hypothesis test such as on this page. Is this correct? How would I set it up for my case?

  • I can adjust my sample size, how do I determine the smallest sample size I would need for such a test.

  • $\begingroup$ A statement like the ones you suggested doesn't necessarily result from hypothesis testing. You might be better served by a confidence interval. $\endgroup$ – Glen_b -Reinstate Monica Nov 11 '13 at 16:10

This is a classic ball-and-urn problem. Most ball-and-urn problems can be associated with a particular probability distribution; the correct choice of pmf for a given problem depends upon the specific details of that problem but here is a list of some of the more common ones. The correct pmf to use for this problem is the hypergeometric distribution. In the notation of the wikipedia page provided in the link, $N=100000$, while $K$ signifies an unknown number of "white balls" which represent a positive outcome among the 100000 records, $(N-K)$ signifies the number of "black balls" which represent a negative outcome, $n$ represents the size of the much smaller $(n \ll N)$ random subsample that is being used for estimation purposes, and $k$ represents the number of white balls actually observed in the $n$-sized subsample. From the definition of the hypergeometric distribution, the probability of drawing $k$ white balls is $$p(k|K,n,N) = \frac{\left(\begin{array}{c} K \\ k \end{array} \right) \left(\begin{array}{c} N-K \\ n-k \end{array} \right)}{\left(\begin{array}{c} N \\ n \end{array} \right)}$$ For this problem, $k$, $n$, and $N$ are known, and $K$ is the unknown quantity about which we wish to perform statistical reasoning. Evidently we need to "invert" the above pmf somehow to find $p(K|k,n,N)$.

We can obtain this "inverse" (so to speak) distribution by applying Bayes rule: $$p(H_{i}|D,B) = \frac{p(D|H_{i},B)p(H_{i}|B)}{p(D,B)} = \frac{p(D|H_{i},B)p(H_{i}|B)}{\sum\limits_{j} p(D|H_{j},B)p(H_{j}|B)}$$ In the Bayes rule notation, $D$ represents the data, i.e., the known observation $k$, while $B$ signifies background information, such as $N$ and $n$. The $H_{i}$ represent a set of mutually exclusive hypotheses about the unknown quantity $K$. For example, $H_{0}$ represents the hypothesis that $K=0$, $H_{1}$, the hypothesis that $K=1$, etc. Thus, we see that $p(D|H_{i},B)$ is equivalent to $p(k|K,n,N)$, the expression already provided above. Bayesian theory requires that we define a so-called prior distribution, represented in the Bayes rule expression above by the term $p(H_{i}|B)$. Bayesian theory provides us some latitude in how to choose the prior, meaning that there isn't necessarily a single correct choice, only that our choice should be reasonable and justifiable. The problem statement specifies that $K$ is known to be not all that large; i.e., $(K \ll N)$. So, let's imagine that we can specify some $K_{max}$, a maximum plausible value for $K$ such that we are sure that $K < K_{max}$, although we don't know anything more for sure a priori about $K$ other than that. Then we may select the prior probability for each of the hypotheses $H_{i}$ as $p(H_{i}|B) = \frac{1}{K_{max}}$. For example, if we were to assume that the parent sample $N$ contained no more than 1% positives, then we would choose $K_{max} = 1000$, and set $$\begin{array}{ccc} p(H_{0}|B) & = & \frac{1}{1000} \\ p(H_{1}|B) & = & \frac{1}{1000} \\ & \vdots & \\ p(H_{999}|B) & = & \frac{1}{1000} \end{array}$$ with $p(H_{1000}|B) = 0$ and similar for all $i \ge 1000$.

Plugging all of this into Bayes rule, we find that $$P(K|k,n,N) = \frac{p(k|K,n,N)}{\sum\limits_{j=0}^{(K_{max}-1)} p(k|K,n,N)}$$ (note that the prior probabilities ultimately all cancel). This expression can be used to answer statistical questions about $K$. For example, to calculate the probability that $K=0$, one literally just plugs in $K=0$, plus the known values for $k$, $n$ and $N$. To calculate a 95% confidence limit on the maximum size of $K$, sum over the individual probabilities for $K=0$, $K=1$, $K=2$, etc., and simply keep increasing $K$ and adding to the sum until you reach some value such that the sum over all of the individual probabilities is 95%. (In practice, you may need to use Stirling's approximation in order to estimate some of the factorials that you will need to calculate.)

One last point: the "answer" that you get for your statistical estimate of $K$ will depend strongly on $n$, the size of the subsample. This makes sense, intuitively: if $n$ is large, say $n=50000$, or in other words half the size of the parent sample of $N=100000$, you will be giving yourself a lot more information to work with than if $n=50$, and that difference will have a profound effect on how much you think you "know" (in a statistical sense) about $K$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.