2
$\begingroup$

I'm struggling to reconcile some minimum sample size calculations. I'm seeking a minimum sample size for two-sample two-tailed proportion z test. I set my parameters as follow...

  • alpha = 0.05
  • power = 0.80
  • kappa = 1
  • pA = 0.45
  • pB = 0.55

...and get a minimum sample size of nA = 389 and nB = 389. See here. So far so good.

But when I run a two-sample two-tailed proportion test using these resulting parameters:

  • pA = 0.45
  • pB = 0.55
  • nA = 389
  • nB = 389

...I get a p-value of 0.005. See here.

The problem is that I think the small p-value suggest the model is greatly over-powered, meaning I could make due with a much smaller sample size. I believe the first calculation suggests a test with these sample sizes / parameters would give me a p-value closer to alpha=0.05. In fact, I can achieve the p-value of 0.048 that I expected by using half the sample size suggested in the minimum sample size equation(nA = 195, nB = 195).

Is nB=389 and nA=389 actually the minimum sample size needed, if so why am I getting such a small p-value, and what am I doing wrong?

Thanks so much!

$\endgroup$

2 Answers 2

5
$\begingroup$

I think the confusion arises because the premise of your question is flawed. In particular, as I understand it, you seem to think there is a discrepancy because you seem to be intuiting a notion that the minimum sample size calculation and the two-sample two-tailed proportion test are somehow supposed to be "inverses" of one another. This intuition (if it's the intuition that you really had) is false, and defining concepts more precisely might help see why.

When you are doing a sample size calculation, the precise question you are asking is as follows: "If the data are generated such that group $A$ has population proportion $p_A$ while group $B$ has population proportion $p_B$, and assuming I collect data in proportion $\kappa$, how large does my sample need to be so that I have a probability $\beta$ of rejecting that the two samples have the same mean at the $\alpha$ significance level?" The key point to notice is that in order to reject with a high probability like $\beta=0.8$, the mean of the sampling distribution needs to be more than $q_{1-\alpha/2}$ standard errors away from 0. This is because when you draw from actual data, you will sometimes (in fact, roughly half the time!) get "unlucky" and draw data where the difference between the sample proportions are closer together than the population proportions. As a result, you need your sample size to be big enough so that even most of the time when you're unlucky in this way, your standard errors are sufficiently low so that you will still be far enough away from equality that your statistical test would reject the null that the two proportions are equal.

Now consider the two-proportion test. The question that the test is asking is "if my data happened to be such that a fraction, $p_A$, of the data in group $A$ was positive and a fraction $p_B$ of group $B$ was positive, and sample sizes are $n_A, n_B$ respectively, what is the probability that such an extreme deviation occurred under the null hypothesis that in the population, the two samples came from the same population, with the same underlying variance."

Hopefully, stating the definitions of these two calculations has made it clear that the two calculations are not "opposites" or "inverses" of one another, so there is no reason why plugging the outputs from one calculations into the inputs for another should return back the inputs from the original calculation. In your specific case, the intuition for why is that in order to have an 80% chance of getting statistical significance, you need to give yourself enough "wiggle room" so that even if the difference in proportions in sample happens to less than the population (which it will in practice half of the time, just from sampling variation), it is still a sufficiently large difference most of the time that you will still reject at significance level 5%. The fact that you are giving yourself this wiggle room means in the average case (which is what you plugged into your two-sample proportion test as inputs), you should be comfortably rejecting the null hypothesis. If you were teetering right on the edge, like in your proposal, you would only find statistical significance 50% of the time (in the 50% of cases where just by dumb luck, the difference in proportions is less than the population mean, you would fail to reject), which is far less than the 80% you would have wanted.

$\endgroup$
3
  • $\begingroup$ Thanks so much, this is a really thorough helpful explanation! I think I get it. Or at least 80% of it ;) I'll read it through couple more times to be sure. $\endgroup$
    – Kees
    Commented Jul 12, 2022 at 5:51
  • $\begingroup$ Ok stats_model I maybe found a way to help me understand. Here is what I'm wondering: say I generate two random normal dichotomous distributions with means 0.45 and .55 (call them populations A and B) and I draw from each a random sample of n=389, and I run the 2-sample 2-tail z-test on the two samples. I may get a p-value of 0.005 and reject the null. Ok, now say I iterate this 1000 times, should I expect to reject the null x percent of times? (e.g 80% or 95%) Or if I'm still looking at this the wrong way, is there another way I could structure/model this experiment to see it more clearly? $\endgroup$
    – Kees
    Commented Jul 12, 2022 at 11:06
  • 1
    $\begingroup$ Yes exactly. The n=389 is basically saying that if you draw two random normal distributions with means respectively $\mathcal N(0.45, 0.45\cdot0.55)$ and $\mathcal N(0.55, 0.45\cdot0.55)$, and you had $n=389$ observations in each sample you would reject the null of equality $\beta=80\%$ of the time at the $\alpha = 5\%$ significance level $\endgroup$ Commented Jul 12, 2022 at 16:27
3
$\begingroup$

It's best to think of this in terms of sample size needed to achieve a certain precision (margin of error; half the width of a confidence interval) in estimating the difference in two probabilities. As detailed here it requires $n=96$ to estimate a single probability to within a margin of error (at the 0.95 level) of $\pm 0.1$, and 196 observations to estimate the difference to within $\pm 0.1$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.