Why does R power.prop.test & R prop.test show inconsistent results?

I am a bit confused by the differences I am seeing between R's power.prop.test & prop.test functions.

To estimate the sample size needed for an A/B test I am conducting I am using the following function:

> power.prop.test(n=NULL, p1=.01, p2=.02, sig.level=.05, power=.8, alternative='two.sided')

Two-sample comparison of proportions power calculation

n = 2318.165
p1 = 0.01
p2 = 0.02
sig.level = 0.05
power = 0.8
alternative = two.sided

NOTE: n is number in *each* group

The way I am interpreting this is that in order to detect a change of at least 1% absolute, I would need at least 2318 samples in each group.

Now when I use the prop.test function below, I see a discrepency.

> prop.test(x=c(20, 40), n=c(2000, 2000), correct=FALSE, alternative='two.sided', conf.level=.95)

2-sample test for equality of proportions without continuity correction

data:  c(20, 40) out of c(2000, 2000)
X-squared = 6.7682, df = 1, p-value = 0.00928
alternative hypothesis: two.sided
95 percent confidence interval:
-0.017527385 -0.002472615
sample estimates:
prop 1 prop 2
0.01   0.02

Here it seems that with only 2000 observations in each bucket a significant 1% change has been detected at the 95% confidence level. This seems to directly contradict the power.prop.test results that said at least 2318 observations were needed to detect a change of that size.

Why is this discrepancy happening?

Your example ignores power considerations. For doing so, you would need to draw many samples from the hypothezised distributions, not just one.

If we simulate 10'000 samples with only $n_1 = 2000$, then we find signficant results only in 75% of all cases at a level of 5%. With $n_1 = 2318$, we reach the expected 80%.

################################
# Example with only n1 = 2000
################################
n1 <- 2000
B <- 10000
pvalues <- numeric(B)

set.seed(3)
x1 <- rbinom(B, n1, 0.01)
x2 <- rbinom(B, n1, 0.02)

for (i in seq_len(B)) {
pvalues[i] <- prop.test(x=c(x1[i], x2[i]), n=c(n1, n1), correct=FALSE, alternative='two.sided', conf.level=.95)$p.value } # Power mean(pvalues <= 0.05) # 0.7449 ################################ # Example with n1 = 2318 ################################ n1 <- 2318 B <- 10000 pvalues <- numeric(B) set.seed(3) x1 <- rbinom(B, n1, 0.01) x2 <- rbinom(B, n1, 0.02) for (i in seq_len(B)) { pvalues[i] <- prop.test(x=c(x1[i], x2[i]), n=c(n1, n1), correct=FALSE, alternative='two.sided', conf.level=.95)$p.value
}

# Power
mean(pvalues <= 0.05) # 0.8091

NOTE ABOUT THE CODE: This example uses purposely an old school for-loop instead of apply to increase readability.