1
$\begingroup$

Suppose I estimate a sample size needed for a hypothesis test of proportion difference, using a binomial distributional assumption, with power 80% and effect size 25%.

Then I perform an A/B experiment and conduct the analysis and find a p-value < 0.05, but effect size less then 25%.

Does it means that my test is not statistically significant?

$\endgroup$
  • $\begingroup$ You are using power, $p$-value, and effect size to calculate a volume. I notice that you don't need power for statistical significance afterward. $\endgroup$ – Kitter Catter May 13 at 21:13
2
$\begingroup$

On the contrary: you achieved a significant result despite decreased power. Is it surprising? Suppose you made a bet chancing on an 80% win probability, you win, then you learn the probability was 70%. It can even be the case that, despite a smaller effect size you had increased power. This owes to some form of misspecification in the original power calculation.

Smaller effect size / increased power is not atypical in the setting of dispersed data, such as the T-test, where the variance is often misspecified. For a binomial model, it's a little less typical because the variance depends on the mean, but a similar result happens when the frequency of the outcome in the control group is misspecified. The power of the binomial test depends on the number of events in that group and the type of effect that is being tested. Unspecified correlation causes a form of dispersion as well.

One should be appropriately cautious using any power calculation software which claims to give the power of a "proportions test" using the effect size and sample size alone. Typically, these tests assume there is a 50% event frequency in the control group, and that's usually not the case.

The formula for the effect size of a proportions test is:

$$ h = 2 \arcsin(\sqrt{p_1}) - \arcsin(\sqrt{p_2})$$

so for instance, in Cohen's "Statistical power analysis for the behavioral sciences" in a design where $p_1 = 0.5$ and $p_2 = 0.4$ produces an effect size of $\approx 0.20$. But suppose for instance, the frequency of all events was 10% lower than expected, the effect size is now $\approx 0.21$ and the power for a sample of size 60 per arm goes from 34.4% to 37.0%.

$\endgroup$
0
$\begingroup$

If I understand your question you did a power calculation for a proportions test. Something like:

l <- 0.5
power <- 0.8
siglevel <- 0.05
prop_val <- 1.25
power.prop.test(p1 = l, p2 = prop_val*l, sig.level = siglevel, power = power)
#     Two-sample comparison of proportions power calculation 
#
#              n = 246.0575
#             p1 = 0.5
#             p2 = 0.625
#      sig.level = 0.05
#          power = 0.8
#    alternative = two.sided
#
# NOTE: n is number in *each* group

You then run the experiment with say 246 samples in each group only to find that the second proportion was less than the 1.25 times that you originally expected. There's still a p-value < 0.05.

prop_val <- 1.2
p1 <- 0.5
p2 <- p1*prop_val
n  <- 246
siglevel <- 0.05
prop.test(x = c(p1*n, p2*n), n = c(n, n), conf.level = 1-siglevel)

#   2-sample test for equality of proportions with continuity correction
#
# data:  c(p1 * n, p2 * n) out of c(n, n)
# X-squared = 4.5739, df = 1, p-value = 0.03246
# alternative hypothesis: two.sided
# 95 percent confidence interval:
#  -0.191538958 -0.008461042
# sample estimates:
# prop 1 prop 2 
#    0.5    0.6 

the p-value is clearly less than 0.05, but the relative proportions is also less than what we used to power our test. What happened?

Let's take a look at the process we used to generate/model this data:

num_trials <- 100000
prop_val <- 1.25
num_per_trial <- 246
base_rate <- 0.5
d1 <- rbinom(num_trials, num_per_trial, base_rate)
d2 <- rbinom(num_trials, num_per_trial, base_rate*prop_val)
mean(d2/d1<1.25)
# 0.49

About half the time we should expect that the proportion we observe will be less than the proportion that was used to generate the data.

This is what the power is trying to protect against.

mean(
sapply(1:num_trials,
FUN = function(x) 
prop.test(c(d1[x],d2[x]),c(num_per_trial, num_per_trial))$p.value<siglevel)
)
# 0.78
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.