How do I determine, pre-experiment, the minimum size of a control group that would be needed to satisfy the requirements of a two-proportion z-test?

  • $p_i=\frac{k_i}{n_i}$ is the proportion of successes, $k_i$, in group $i$ which is of size $n_i$. This will only be known post-experiment.
  • The size of the test group, $n_1$ is known. The size of the control group, $n_2$, can be chosen to satisfy the statistical validity requirements of the two-proportion z-test.
  • The standard deviation of both groups is unknown.
  • The null hypothesis is $H_0: p_1 = p_2$. The alternative hypothesis is $H_a : p_1 > p_2$.
  • We wish to reject $H_0$ with probability $\alpha$ when $H_0$ is true (i.e. a p-value of $\alpha$).
  • We wish to reject $H_0$ with a probability of at least $1-\beta$ when $H_a$ is true (i.e. a power of $1-\beta$).
  • We wish to be able to measure an effect of size $x = \frac{p_1}{p_2} - 1$ with power $1-\beta$ and p-value $\alpha$.

How can I determine the minimum required value of $n_2$ such that after my experiment I can run a two-proportion z-test that satisfies the above criteria? I have used generic variables hoping that a formulaic answer exists that can use these variables.

  • $\begingroup$ 1. In your second bullet point do you mean $n_1$ where you have $n_i$? ... 2. Hypotheses are about populations not sample estimates -- your hypotheses should not have hats. ... 3. note that the power depends on how different the two population proportions are. If the difference is large the power will be bigger than if it is small. You need to specify what difference you want power $\beta$ at. .... 4. Your understanding is not correct -- there's currently no reason to believe that $n_2=n_1$ will do what you want. ... please edit your question to clarify / add details as needed $\endgroup$ – Glen_b May 17 '17 at 8:10
  • $\begingroup$ Thanks @Glen_b, I have updated the question to address your comment. $\endgroup$ – conor May 17 '17 at 8:26
  • 1
    $\begingroup$ You haven't stated what population difference you want power $\beta$ at -- you have only said that you estimate the difference will be something. This is a somewhat different proposition; I might think one proportion is 20% bigger than another, for example, but want power to be 90% in the case when it's 50% bigger -- you should make it explicit if this is the difference you want power $\beta$ at $\endgroup$ – Glen_b May 17 '17 at 8:41
  • $\begingroup$ Sorry, I want the power to be $\beta$ in the case that $p_1$ is $x\%$ greater than $p_2$. Does that clarify what I'm asking / do I make sense? As a side question, if it later turns out that the difference is larger, that is the effect size is larger, is it correct to say that all else being equal, the power to detect that larger effect size has increased? Or would one say the power to detect the original effect size $x$ has increased? Or neither? $\endgroup$ – conor May 18 '17 at 2:30
  • $\begingroup$ Yes, I think you've covered everything (though it should be in your question too). If the population effect size is larger than the one you worked out the sample size from, you'd generally have more power than that specified (in particular, this is certainly true to the extent that the normal approximation being used here is accurate). $\endgroup$ – Glen_b May 18 '17 at 3:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.