sample size calculated with power.prop.test cs. sample size for significance in chisq.test?

Question

I want to calculate the sample size for my AB-Test with the power.prop.test() in R. Let's say I expect an uplift in an acceptance rate from 0.33% to 0.34% and want to know how big my sample size has to be. I use the power.prop.test()

power.prop.test(n = NULL,
               p1 = 0.33, 
               p2 = 0.34, 
               sig.level = 0.05, 
               power = 0.8, 
               alternative = c("two.sided"),
               strict=T)

and get

Two-sample comparison of proportions power calculation 

          n = 34969.42
         p1 = 0.33
         p2 = 0.34
  sig.level = 0.05
      power = 0.8
alternative = two.sided

NOTE: n is number in *each* group

So I would need 34970 cases in each group. When I just use half of it - so 17485 cases - and the corresponding acceptance rates of 0.33 and 0.34 I get the following:

d1<-data.frame("acceptance"=c(17485*0.33,  17485*0.34),
               "no acceptance"=c(17485-17485*0.33,  17485-17485*0.34))
chisq.test(d1,correct=F)

which gives the following result:

Pearson's Chi-squared test with Yates' continuity correction

data:  d1
X-squared = 3.8863, df = 1, p-value = 0.04868

So it is significant, although I only took half of the sample size... What did I get wrong here? Thanks for your help!

Answer 1

You're misunderstanding what the power analysis is telling you: that when the true, unobserved, probabilities of acceptance are 0.33 & 0.34, an experiment with a sample size of 34970 in each group has an 80% chance (approximately) of resulting in a test-statistic significant at the 5% level. So the way to check this is to simulate the data-generating process—in each simulation the estimated, observed probabilities will differ somewhat from 0.33 & 0.34. Try out the following function:

calc.power <- function(p1, p2, n, alpha, no.simulns){
  p.values <- numeric(no.simulns)# make a vector of p-values
  for (i in 1:no.simulns){
    rbinom(1, n, p1) -> no.accepted.1 # simulate for group 1 by drawing from binomial distribution
    rbinom(1, n, p2) -> no.accepted.2 # simulate again for 2nd group
    d1 <- data.frame("acceptance"=c(no.accepted.1,  no.accepted.2),
                     "no acceptance"=c(n-no.accepted.1,  n - no.accepted.2))
    chisq.test(d1, correct=F)$p.value -> p.values[i] # perform Pearson's chi-squared test on simulated data
  }
  sum(p.values <= alpha)/no.simulns # calculate proportion of tests significant at specified alpha level
}