# Testing count post sampling

Some automated sampling is done (by a computer) on a collection of $N$ identical items. The sampling process consists in keeping each item with a constant probability $\alpha$ and discarding it with a probability $1-\alpha$. Call $X$ the number of items that have been kept at the end of the process.

The expected value of $X$ is $\alpha N$. More precisely, $X$ has a binomial distribution $B(N,\alpha)$ given $N$. Most often $N$ is large enough to use a normal approximation $B(N,\alpha)\approx\mathcal{N}(\alpha N,\alpha(1-\alpha)N)$.

The data are events recorded on the Internet. $N$ is from a few hundreds to a several millions. I know the sampling rate is exactly $\alpha=\frac{1}{10}$.

Now two sampling processes are done independently from two populations $N_A$ and $N_B$ with the same known sampling rate $\alpha=\frac{1}{10}$. After sampling we have $X_A$ and $X_B$ items being kept respectively. For example, we may get something like: $X_A=1234, X_B=1322$. I want to test if the original populations are the same: I want to test the null hypothesis $N_A=N_B$.

How can I do ?

All I can observe is $X_A$ and $X_B$. I know for sure the sampling processes are independent (I'm not even sure this is important). I know the rate $\alpha$ is perfectly known.

Unlike most tests I've heard of (Kolmogorov-Smirnov, Fisher, $\chi^2$...) this is a test on raw counts and not on proportions (probabilities). That's why I struggle to find a standard technique.

• Could you please explain what "X items left" means? Is it the count of items in the sample, as suggested by your distributional assertions? And shall we also deduce that "sampling" means each item is independently selected with a constant probability of $\alpha$? – whuber May 31 '17 at 2:11
• Yes. And thanks for your comment. I edited the question to make all these points clearer. – Benoit Sanchez May 31 '17 at 12:56

I have found a solution that is satisfying for me.

Even though it's not a $\chi^2$ test we can use a method very close to a $\chi^2$ test. Let's define :

$$S=\frac{(X_A-X_B)^2}{(1-\alpha)(X_A+X_B)}$$

Assume the null hypothesis is true that is : $N_A=N_B$. Simply write them $N$. If $N$ is large enough, we can use approximations :

First about the numerator. $X_A$ has a normal distribution $\mathcal{N}(\alpha N,\alpha(1-\alpha)N)$. Same for $X_B$. Since $X_A$ and $X_B$ are independent, $X_A-X_B$ has a normal distribution $\mathcal{N}(0,2\alpha(1-\alpha)N)$. It is $\sqrt{2\alpha(1-\alpha)N}$ multiplied by a variable having a distribution $\mathcal{N}(0,1)$. Thus $(X_A-X_B)^2$ is $2\alpha(1-\alpha)N$ multiplied by a variable having a $\chi^2$ distribution with one degree of freedom.

For the denominator, I used a rough approximation. Because of the law of large numbers, $X_A+X_B$ is approximatively a constant $2\alpha N$. Note that I could have used $2X_A$ or $2X_B$ instead of $X_A+X_B$ but I wanted something symmetrical. Finally the denominator is approximated by $2\alpha(1-\alpha)N$. I guess this approximation needs $N$ to be larger than for the first approximation, but I don't really know.

Thus, when you assume the null hypothesis, $S$ has a $\chi^2$ distribution with $1$ degree of freedom.

I can use a classical $p$-value from the $\chi^2$ table or this nice calculator : http://www.danielsoper.com/statcalc/calculator.aspx?id=11. With my example $X_A=1234, X_B=1322$ I get $S=3.36$ and $p=0.065$. Under the null hypothesis, such a difference can be observed with probability 6.5%. This does not lead to reject the null hypothesis with usual limits (say 1% or 5%).

I ran this test on the data, which was actually hundreds of thousands of population pairs : hundred of thousands of these tests. Thus I chose a small limit for the $p$-value : around $10^{-6}$. I found flawed populations (null hypothesis rejected) leading to identify true bugs in the event recording process : $p$-values for buggy population pairs were actually extremely small : smaller than $10^{-20}$. After fixing them, all the tests became green (null hypothesis accepted).

In your example, $X_A = 1234$. No doubt after 1370 items, you will take 137 (1/10 of 1370) as sample, and 1233 were left. Next time, if you wait until other 10 items ready to take another one as sample, then $X_A = 1234$ means after you take 137th item for sample 1 more item comes in and you are waiting. So $N_A = 1370 + 1 = 1371$. If you do not waiting for 10 items and take the first one after 1370 items, then $N_A = 1370 + 1 (\mathrm{sample}) + 1 (\mathrm{left}) = 1372$. Anyway $N_A = 1371$ or $1372$ and no third value.

Similarly $X_B = 1322 \rightarrow N_B = 1460 + 8(\mathrm{left }) (+1(\mathrm{sample})) = 1468$ or $1469$, where (+1) mean may or may not add 1, see above.

Conclusion: Reject null hypothesis that $N_A = N_B$ because $N_A = 1371$ or $1372$ and $N_B = 1468$ or $1469$ and there is no overlap.

• This obviously is not a hypothesis test, and your expressions for the counts $N_A$ and $N_B$ are not integral, so it's doubtful that it correctly answers the question. Could you explain how you're interpreting the problem? – whuber May 31 '17 at 2:12
• Yes. Want I meant is what whuber understood. The sampling is based on random independent trials. I updated the question to make it clearer. – Benoit Sanchez May 31 '17 at 13:01