2
$\begingroup$

Suppose I have a two phase experiment. The goal of the experiment will be to test if there are differences in proportions between two treatments. In phase one, I have no idea how many samples I will need as I have no prior information, so say I take 30 samples for each treatment and get proportions of 0.5 and 0.6.

Now I want to use this information to calculate the number of samples I will need in phase two in order to have an 80% chance (power) to show a difference between the two treatments (assume alpha = 0.05). The sample size calculators I have found online do not apply as you to enter in the population proportions, but I don't have the population proportions, only estimated proportions based on the first sample. So I need to take the sample variation into account somehow in the sample size calculation.

Any help would be greatly appreciated.

$\endgroup$

2 Answers 2

1
$\begingroup$

In various statistical software programs (and, allegedly, in some online 'calculators') you can specify typical proportions that you'd like to be able to distinguish at the 5% level of significance and with power 80%.

Specifically, if reasonable proportions for Treatments 1 and 2 are $p_1 = 0.5$ and $p_2 = 0.6,$ then these are the 'proportions' you enter. (Of course, you won't know the exact proportions, but the difference between them should be the size of difference you'd like to be able to detect.)

Sample size computation from Minitab. In particular, output from a 'power and sample size' procedure in a recent release of Minitab is shown below. For a two-sided test with the proportions guessed above, you'd need $n=388$ in each group for 80% power.

Power and Sample Size 

Test for Two Proportions

Testing comparison p = baseline p (versus ≠)
Calculating power for baseline p = 0.5
α = 0.05


              Sample  Target
Comparison p    Size   Power  Actual Power
         0.6     388     0.8      0.800672

The sample size is for each group.

enter image description here

Often tests to distinguish between two binomial proportions are done in terms of approximate normal tests, which are quite accurate for sample sizes this large and for success probabilities not too near to $0$ or $1.$

Example of test of two proportions. Suppose that your results are $183$ in the first group and $241$ in the second. Then Minitab's version of the one-sided test shows a highly significant difference with a P-value near $0.$

Test and CI for Two Proportions 

Sample    X    N  Sample p
1       182  388  0.469072
2       241  388  0.621134

Difference = p (1) - p (2)
Estimate for difference:  -0.152062
95% CI for difference:  (-0.221312, -0.0828117)
Test for difference = 0 (vs ≠ 0):  
  Z = -4.30  P-Value = 0.000

Similar test in R: For comparison, the version of the test implemented in the R procedure 'prop.test' gives the following result, also leading to rejection of the null hypothesis. (I use the version without continuity correction on account of the large sample size.)

prop.test(c(182,241), c(388,388), cor=F)

       2-sample test for equality of proportions 
       without continuity correction

data:  c(182, 241) out of c(388, 388)
X-squared = 18.091, df = 1, p-value = 2.106e-05
alternative hypothesis: two.sided
95 percent confidence interval:
 -0.22131203 -0.08281168
sample estimates:
   prop 1    prop 2 
0.4690722 0.6211340 

Simulation of power. The following simulation in R with 'prop.test' shows that the power of the test to distinguish between proportions $0.5$ and $0.6$ at the 5% level is roughly 80%.

set.seed(112)
pv = replicate(10^5, prop.test(rbinom(2,388,c(.5,.6)),c(388,388),cor=F)$p.val)
mean(pv <= .05)
[1] 0.79673
$\endgroup$
4
  • $\begingroup$ I guess what I'm still confused about is this: Case 1: In the first phase I get two proportions: 6/10 = 0.6, and 5/10 = 0.5. Case 2: In the first phase I get two proportions: 60/100 = 0.6 and 50/100 = 0.5. Will my following sample size need to be larger under case 1 since there is more uncertainty about the estimated proportions? $\endgroup$
    – DanE
    Nov 3, 2020 at 18:27
  • 1
    $\begingroup$ Suppose all you know about the success probabilities is from Phase 1: .5 and .6 for twotreatments, That gives info that success probabilities are near 1/2 and that you might want to detect a difference of about .1 in success probabilities. Unfortunately, you need specific inputs to a power and sample size computation--even if only educated guesses. If you have better guesses than what you have not revealed, then of course you will want to use the best guesses you have as you plan sample sizes for phase 2. Also, in the real world, regulatory agencies may demand a certain sample size. $\endgroup$
    – BruceET
    Nov 3, 2020 at 18:40
  • $\begingroup$ I see, so if my understanding is correct the estimates from phase 1 are, at the end of the day, your best guesses for what the proportions might be. More importantly, the uncertainty of the estimates is irrelevant since you don't know which way they might vary and the distribution is normal around the estimates. $\endgroup$
    – DanE
    Nov 3, 2020 at 19:19
  • $\begingroup$ The uncertainty (do you mean variance?) of estimates of binomial proportions is not irrelevant, but determined by $n$ and $p.$ // You need to focus on what you can guess directly from Phase 1 or from any previous similar trials (success probability small, medium, or large? Seems medium here.) and how large a difference btw success probabilities for the two treatments is of practical importance. (Is difference btw success probabilities .5 and .6 of clinical importance?) $\endgroup$
    – BruceET
    Nov 3, 2020 at 20:38
0
$\begingroup$

I believe the article below accurately answers the problem:

https://www.bmj.com/content/bmj/306/6886/1181.full.pdf

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.