Imagining the following situation where A/B are independent results:

  • A is a 51% result, given N datapoints
  • B is a 50% result, given M datapoints

how many data points (ie N and M) do you need to have confidence that the difference between A/B is statistically significant?


There are various sample size calculators you can use online and in various software for a two-sample proportions test.

Here's the calculation using Stata that suggests that you would need N=M=39,240 for a two-sided alternative test at conventional levels:

. power twoproportions .50 .51

Performing iteration ...

Estimated sample sizes for a two-sample proportions test
Pearson's chi-squared test 
Ho: p2 = p1  versus  Ha: p2 != p1

Study parameters:

        alpha =    0.0500
        power =    0.8000
        delta =    0.0100  (difference)
           p1 =    0.5000
           p2 =    0.5100

Estimated sample sizes:

            N =    78,480
  N per group =    39,240
| cite | improve this answer | |
  • $\begingroup$ @enderland Did this clarify things? $\endgroup$ – Dimitriy V. Masterov Feb 5 '18 at 22:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.