Testing count post sampling

Question

Some automated sampling is done (by a computer) on a collection of $N$ identical items. The sampling process consists in keeping each item with a constant probability $\alpha$ and discarding it with a probability $1-\alpha$. Call $X$ the number of items that have been kept at the end of the process.

The expected value of $X$ is $\alpha N$. More precisely, $X$ has a binomial distribution $B(N,\alpha)$ given $N$. Most often $N$ is large enough to use a normal approximation $B(N,\alpha)\approx\mathcal{N}(\alpha N,\alpha(1-\alpha)N)$.

The data are events recorded on the Internet. $N$ is from a few hundreds to a several millions. I know the sampling rate is exactly $\alpha=\frac{1}{10}$.

Now two sampling processes are done independently from two populations $N_A$ and $N_B$ with the same known sampling rate $\alpha=\frac{1}{10}$. After sampling we have $X_A$ and $X_B$ items being kept respectively. For example, we may get something like: $X_A=1234, X_B=1322$. I want to test if the original populations are the same: I want to test the null hypothesis $N_A=N_B$.

How can I do ?

All I can observe is $X_A$ and $X_B$. I know for sure the sampling processes are independent (I'm not even sure this is important). I know the rate $\alpha$ is perfectly known.

Unlike most tests I've heard of (Kolmogorov-Smirnov, Fisher, $\chi^2$...) this is a test on raw counts and not on proportions (probabilities). That's why I struggle to find a standard technique.

Answer 1

I have found a solution that is satisfying for me.

Even though it's not a $\chi^2$ test we can use a method very close to a $\chi^2$ test. Let's define :

$$S=\frac{(X_A-X_B)^2}{(1-\alpha)(X_A+X_B)}$$

Assume the null hypothesis is true that is : $N_A=N_B$. Simply write them $N$. If $N$ is large enough, we can use approximations :

First about the numerator. $X_A$ has a normal distribution $\mathcal{N}(\alpha N,\alpha(1-\alpha)N)$. Same for $X_B$. Since $X_A$ and $X_B$ are independent, $X_A-X_B$ has a normal distribution $\mathcal{N}(0,2\alpha(1-\alpha)N)$. It is $\sqrt{2\alpha(1-\alpha)N}$ multiplied by a variable having a distribution $\mathcal{N}(0,1)$. Thus $(X_A-X_B)^2$ is $2\alpha(1-\alpha)N$ multiplied by a variable having a $\chi^2$ distribution with one degree of freedom.

For the denominator, I used a rough approximation. Because of the law of large numbers, $X_A+X_B$ is approximatively a constant $2\alpha N$. Note that I could have used $2X_A$ or $2X_B$ instead of $X_A+X_B$ but I wanted something symmetrical. Finally the denominator is approximated by $2\alpha(1-\alpha)N$. I guess this approximation needs $N$ to be larger than for the first approximation, but I don't really know.

Thus, when you assume the null hypothesis, $S$ has a $\chi^2$ distribution with $1$ degree of freedom.

I can use a classical $p$-value from the $\chi^2$ table or this nice calculator : http://www.danielsoper.com/statcalc/calculator.aspx?id=11. With my example $X_A=1234, X_B=1322$ I get $S=3.36$ and $p=0.065$. Under the null hypothesis, such a difference can be observed with probability 6.5%. This does not lead to reject the null hypothesis with usual limits (say 1% or 5%).

I ran this test on the data, which was actually hundreds of thousands of population pairs : hundred of thousands of these tests. Thus I chose a small limit for the $p$-value : around $10^{-6}$. I found flawed populations (null hypothesis rejected) leading to identify true bugs in the event recording process : $p$-values for buggy population pairs were actually extremely small : smaller than $10^{-20}$. After fixing them, all the tests became green (null hypothesis accepted).

Answer 2

In your example, $X_A = 1234$. No doubt after 1370 items, you will take 137 (1/10 of 1370) as sample, and 1233 were left. Next time, if you wait until other 10 items ready to take another one as sample, then $X_A = 1234$ means after you take 137th item for sample 1 more item comes in and you are waiting. So $N_A = 1370 + 1 = 1371$. If you do not waiting for 10 items and take the first one after 1370 items, then $N_A = 1370 + 1 (\mathrm{sample}) + 1 (\mathrm{left}) = 1372$. Anyway $N_A = 1371$ or $1372$ and no third value.

Similarly $X_B = 1322 \rightarrow N_B = 1460 + 8(\mathrm{left }) (+1(\mathrm{sample})) = 1468$ or $1469$, where (+1) mean may or may not add 1, see above.

Conclusion: Reject null hypothesis that $N_A = N_B$ because $N_A = 1371$ or $1372$ and $N_B = 1468$ or $1469$ and there is no overlap.