# How to check reliability of a given sampling technology?

Very basic question, I suspect. It's sort of the inverse of this one, and the same disclaimers about my own ignorance apply, but here goes:

I have a sampling technology. I know the total population, and I know the number of samples I've done. What's the best way to calculate and express how confident I am that my results are correct?

A few more points in case they make a difference:

• My test checks whether an item is in one of three states (and they are the only possible states).
• I'm assuming that the test is truly a random distribution.
• In general, my total population is > 1,000,000 and my sample is ~20% of that, but it can vary quite a bit.

And a wrinkle (I hesitate to even put this in because I want to keep things simple):

• There's a slight error where my reported Total Population value might be slightly smaller than the actual total (by 5-10%). I can get the actual total if need be, but won't bother if it doesn't really make a difference.

Rough and simple / clear is probably better than more accurate if there are different ways of expressing this. I.e., something like "we're 95% confident that this is correct" is probably better than "we're 95% confident that this is correct to within 3%". (Am I talking about P?)

UPDATE:

Hmm... I think my "wrinkle" may have introduced confusion. I'm not trying to find out how accurate the total is. I can get the actual total, but I have to use a more convoluted method. So my main question was about saying if I have 10,000 samples out of 1,000,000 items how confident can I be that my sampled distribution among the three states is correct.

My secondary question (which is where the wrinkle comes in) is: given that the actual total is 5-10% larger than the one I'm using, how much of a difference does that make to my confidence? In other words, I'm saying 1,000,000 items, but it might actually be 1,100,000. Should I bother to go through the convoluted process to get the actual total, just to compute the confidence? To me, that seems unlikely to make a significant difference to my confidence level, but I thought I should check.

• you say you have 10,000 samples - I take it this means you have one sample, and it has 10,000 items. Or is it that you have ten thousand samples of unspecified number of units each? I'll update my answer with some more details about how the equations work. Commented Apr 9, 2011 at 1:18

This sounds like a simple case of hyper-geometric sampling. So you have a sampling distribution of:

$$p(r_1,r_2,r_3|R_1,R_2,R_3,I)=\frac{{R_1 \choose r_1}{R_2 \choose r_2}{R_3 \choose r_3}}{{R_1+R_2+R_3 \choose r_1+r_2+r_3}}$$

Capitals letters denote population totals, and small letters denote sampled numbers. $I$ is the prior information about the sampling. You want to "invert" this to get a distribution for $R_{i}$ Just use Bayes theorem:

$$p(R_1,R_2,R_3|r_1,r_2,r_3,I)=p(R_1,R_2,R_3|I)\frac{p(r_1,r_2,r_3|R_1,R_2,R_3,I)}{p(r_1,r_2,r_3|I)}$$

And you now have a statement about the accuracy of the population totals!

UPDATE

In response to the comment, the values of $R_{j}$ are the unknown population totals for each of the three states - so if you had sampled/tested every item, you would get $R_{j}$ as the numbers in each state. I assume that these quantities are the target of your inference - this is what you would like to know.

Now what do you know? the population total $N=R_{1}+R_{2}+R_{3}\approx 1,000,000$ (we can account for possible error later). You also know the sampled numbers $r_{1},r_{2},r_{3}$ (the number which tested positive for each state in your sample). The total total sample size $n=r_{1}+r_{2}+r_{3}$.

the quantity $p(R_1,R_2,R_3|I)$ is called the prior, and you assign it based on what is known about the population totals beyond the data from the sample. Now you have stated that $N$ is known, which constrains but does not determine the prior. One way to determine the prior is to break up the three propositions into mutually exclusive and exhaustive pieces, and assign equal probabilities to the ones which add to $N$, and zero to everything else. A quick counting exercise shows there is $\frac{(N+1)(N+2)}{2}$ combinations of $R_1,R_2,R_3$ which add up to $N$, so the joint prior is:

$$p(R_1,R_2,R_3|N,I)=\frac{2}{(N+1)(N+2)}\delta(N-R_1-R_2-R_3)$$

Where $\delta(x)=1$ if $x=0$ and $\delta(x)=0$ if $x\neq 0$. And you can work out the normalising constant $P(r_1,r_2,r_3|N,I)$ by adding up the prior and the sampling probabilities over the $R_{j}$, so we get:

$$p(r_1,r_2,r_3|N,I)=\sum_{R_1=0}^{N}\sum_{R_2=0}^{N}\sum_{R_3=0}^{N}p(R_1,R_2,R_3|N,I)p(r_1,r_2,r_3|R_1,R_2,R_3,N,I)$$ $$=\sum_{R_1=0}^{N}\sum_{R_2=0}^{N-R_1}\frac{2}{(N+1)(N+2)}\frac{{R_1 \choose r_1}{R_2 \choose r_2}{N-R_1-R_2 \choose n-r_1-r_2}}{{N \choose n}}$$

Now $(N+1)(N+2){N \choose n}=(n+1)(n+2){N+2 \choose n+2}$ and

$$\sum_{R_1=0}^{N}\sum_{R_2=0}^{N-R_1}{R_1 \choose r_1}{R_2 \choose r_2}{N-R_1-R_2 \choose n-r_1-r_2}={N+2 \choose n+2}$$

So we get:

$$p(r_1,r_2,r_3|N,I)=\frac{2}{(n+1)(n+2)}$$

And thus the posterior distribution is:

$$p(R_1,R_2,R_3|r_1,r_2,r_3,N,I)=\frac{2}{(N+1)(N+2)}\frac{\frac{{R_1 \choose r_1}{R_2 \choose r_2}{R_3 \choose r_3}}{{N \choose n}}}{\frac{2}{(n+1)(n+2)}}=\frac{{R_1 \choose r_1}{R_2 \choose r_2}{R_3 \choose r_3}}{{N+2 \choose n+2}}$$

The last form shows very easily how to generalise, for those interested (noting that $2=3-1$). This posterior has expectation for $R_1$ of:

$$E([R_1+1]|r_1,r_2,r_3,N,I)=\sum_{R_1=0}^{N}\sum_{R_2=0}^{N-R_1}\frac{(R_1+1){R_1 \choose r_1}{R_2 \choose r_2}{R_3 \choose r_3}}{{N+2 \choose n+2}}=\frac{(r_1+1)(N+3)}{n+3}$$ $$\implies E(R_1|r_1,r_2,r_3,N,I)=\frac{(r_1+1)(N-n+n+3)-(n+3)}{n+3}$$ $$=r_1+(N-n)\hat{p}$$

where $\hat{p}=\frac{r_1+1}{n+3}$. This is the number of observed in category 1 plus an estimate of the number remaining unobserved in category 1. Now for the accuracy we can take the variance - which, using the same trick calculate $$E([R_1+1][R_1+2])=\frac{(r_1+1)(r_1+2)(N+3)(N+4)}{(n+3)(n+4)}=E([R_1+1]^2)+E(R_1+1)$$ and note that $var(R_1)=var(R_1+1)$ we get

$$var(R_1)=E([R_1+1][R_1+2])-E(R_1+1)-[E(R_1+1)]^2$$ $$=\frac{(r_1+1)(N+3)}{n+3}\left[\frac{(r_1+2)(N+4)}{n+4}-1-\frac{(r_1+1)(N+3)}{n+3}\right]$$ which after some tedious manipulations you get:

$$var(R_1)=\frac{\hat{p}(1-\hat{p})}{n+4}(N-n)(N+3)$$

You could also calculate the mean and variance of the fraction remaining $F=\frac{R_1-r_1}{N-n}$ which are given by:

$$E(F|r_1,r_2,r_3,N,I)=\hat{p}\;\;\;\;\;var(F|r_1,r_2,r_3,N,I)=\frac{\hat{p}(1-\hat{p})}{n+4}\left(1+\frac{n+3}{N-n}\right)$$

And then quantities are approximately independent of $N$ - so the accuracy of $N$ is not important for inferring the proportions in each category - but the sampling fraction is.

One way to incorporate the uncertainty about $N$ is to use a uniform prior between to bounds $L_N<N<U_N$, and then "average out" the value of $N$ from the posterior:

$$p(R_1,R_2,R_3|r_1,r_2,r_3,I)=\frac{1}{U_N-L_N}\sum_{N=L_N}^{U_N}p(R_1,R_2,R_3|r_1,r_2,r_3,N,I)$$

But unless the terms in this summation are appreciably different, the result won't change that much. It won't in this case as I have shown

• I really appreciate the answer (though I'm not quite sure how to plug in my two numbers). I think I may have introduced confusion with my "wrinkle". See my update above. Commented Apr 8, 2011 at 19:34

You might be looking for a goodness-of-fit test to check one of the following

1. whether the sample distribution are likely to have come from the population distribution or
2. whether multiple sample distributions are likely to have come the same unknown population distribution

Look at chi-square and Kolmogorov-Smirnov tests.

I'm not sure what you mean by either "we're 95% confident that this is correct" or "we're 95% confident that this is correct to within 3%". Some questions:

1. What do you want to know about the population: the number of items in the population in each of the three categories, or the proportion of items in each of the three categories?

2. How small would your estimation error need to be to either of these to consider your result "correct"?

3. How do you want to aggregate error across the classes?

I suspect some of these things will be more clear if you provide some additional detail about your application.

• We're measuring something on web page views that can be in one of three states -- A, B & C. Let's say (case 1) there are 1 MIL views, and we sample 100K, and find that 20% were A, 30% were B, and 50% were C. Then for a different page (case 2), we have 1.5 MIL views, but can only sample 5K and get a different set of A/B/C percentages (as expected). I want to express confidence in those percentages in some form. This is for a non-stats audience, so we want to keep the language simple. Something like "we're 99% confident that our measurements are accurate in case 1 and 95% confident in case 2". Commented Aug 26, 2011 at 19:44