I have a finite space of 'data' (rows) that is fairly large (3 billion) and I want to check how many 'Positives' there are in this data set. However, checking the data is time consuming from a computational standpoint. If I have reason to believe that the data is i.i.d and each row has an equal likelihood of being a 'Positive' how can I calculate the sample size required to get a reasonable estimate on the TOTAL NUMBER (not just proportion in a Binomial Distribution) of 'Positives' in the data?

Things I have tried / other caveats:

  • I have looked into Binomial sample size calculation, but I'm not sure how this extends to the case where I am just looking for counts. I am guessing that when I multiply the estimated probability by the 3 billion, that it will distort the confidence intervals as it adds more variance.

  • I have looked into doing a Poisson sample size calculation but can't find many resources online that I can understand.

  • Do I have to assume that data is i.i.d in the distribution of 'Positives'? Intuitively I can understand why it wouldn't matter, since a large enough sample would factor in an aggregate probability of success.


Exploratory solution would be to try a couple of sub-sets of your data, (3000, 30K, 300K, 3m, 30m) and see what happens to your confidence intervals as your data set gets larger. It may be that nothing happens, or it may be that your confidence interval gets larger in a predictable way.

| cite | improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.