0
$\begingroup$

I'm having a hard time wrapping my head around experiment design and what appears to be a disconnect to experiment outcome analysis.

If I am proposing a z test of two independent proportions (conversion rates) where I have an assumed historic conversion rate of p2 = 0.10 and I want to determine the required sample size to detect an effect of 20% (p1 = 0.12) using a one tailed test with alpha = 0.05 and power = 0.80 I would arrive at required sample sizes of roughly 3,025. I am comfortable with this calculation utilizing cohen's h and have confirmed using G*Power as well.

However, If I were to fast forward and assume that I conduct the experiment and indeed my control achieves the assumed historic conversion rate of 0.10 and my treatment achieves my hypothesized conversion rate of 0.12 and I check the p value for that experiment I receive p = 0.006.

Why is this? This makes it seem like I am vastly over sizing my required sample sizes. Am I seeing this effect because my sample sizes are ultimately being inflated to protect against type II errors with 80% confidence?

$\endgroup$
1
  • $\begingroup$ You "fast forward" by assuming that you'll observe $\hat{p}_1 = p_1$ and $\hat{p}_2 = p_2$. However, it's very unlikely that the sample conversion rates will be exactly equal to the population conversion rates. So why pick those two particular $\hat{p}_1$ and $\hat{p}_2$ to compute compute a p-value? $\endgroup$
    – dipetkov
    Sep 23, 2022 at 21:25

1 Answer 1

1
$\begingroup$

I assume your concern here is because the p value is so low that you've gotten too much data, and could have ended the experiment earlier.

Remember that power is a probability (of rejecting the null when the alternative is the case) and as such you should not be concerned with any one result but the results over the long run. Were you to run 100 such experiments, then 80% of them would correctly reject the null. Furthermore, each experiment will be different due to sampling variability; you're getting different people in each experiment, each group acts a little differently than the last. Some will convert a lot more, some will convert only a little more as compared to control.

You can clearly see this if you simulate the experiment

p1 <- 0.1
p2 <- 0.12
n <- 3027

set.seed(0)
pvals <- experiment_results <- replicate(1000, {
  x <- rbinom(2, n, c(p2, p1))
  prop.test(x, rep(n, 2), alternative = 'greater')$p.value
})


hist(pvals)

enter image description here

The p value you obtained is a draw from the distribution this histogram is intended to approximate. That it is so small is by design (quite literally). You've done a good job and have nothing to worry about.

$\endgroup$
4
  • $\begingroup$ I think this makes sense to me. Essentially by conducting the power analysis to minimize the probability of a type II error I'm pushing the design away from the minimum threshold of the experiment. Is that roughly correct? $\endgroup$
    – cjmsmcl
    Sep 23, 2022 at 22:21
  • $\begingroup$ @cjmsmcl First, we don't minimize the probability of a type II error. Don't wan't to make a type II error? Then simply always reject the null. When we compute power, we fix the probability we would reject the null under the assumption the alternative is the case(and all other assumptions are true) ... $\endgroup$ Sep 23, 2022 at 22:29
  • $\begingroup$ ...Now, because of sampling variability, sometimes you will get really large test stats (small p value) and sometimes you will get small test stats (large p values). 80% of those p values should be smaller than your significance value (again, assuming all other assumptions are met). Aside from that, any additional variation is from sampling. You could rerun the experiment and get a larger p value. You've not over sizing, you just got really lucky and saw a big effect. $\endgroup$ Sep 23, 2022 at 22:31
  • $\begingroup$ Perfect, that makes sense. Thank you for the help! $\endgroup$
    – cjmsmcl
    Sep 23, 2022 at 22:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.