I am a bit confused by the differences I am seeing between R's power.prop.test & prop.test functions.
To estimate the sample size needed for an A/B test I am conducting I am using the following function:
> power.prop.test(n=NULL, p1=.01, p2=.02, sig.level=.05, power=.8, alternative='two.sided') Two-sample comparison of proportions power calculation n = 2318.165 p1 = 0.01 p2 = 0.02 sig.level = 0.05 power = 0.8 alternative = two.sided NOTE: n is number in *each* group
The way I am interpreting this is that in order to detect a change of at least 1% absolute, I would need at least 2318 samples in each group.
Now when I use the prop.test function below, I see a discrepency.
> prop.test(x=c(20, 40), n=c(2000, 2000), correct=FALSE, alternative='two.sided', conf.level=.95) 2-sample test for equality of proportions without continuity correction data: c(20, 40) out of c(2000, 2000) X-squared = 6.7682, df = 1, p-value = 0.00928 alternative hypothesis: two.sided 95 percent confidence interval: -0.017527385 -0.002472615 sample estimates: prop 1 prop 2 0.01 0.02
Here it seems that with only 2000 observations in each bucket a significant 1% change has been detected at the 95% confidence level. This seems to directly contradict the power.prop.test results that said at least 2318 observations were needed to detect a change of that size.
Why is this discrepancy happening?