1
$\begingroup$

I want to do an experiment and try to get an initial proportion (e.g., successful rate). The experiment will generate one of two results (either success or failure). I would like to know how many trials are needed to get a successful rate of 0.003 (statistically important) from a statistical point of view? I initially thought that if I did 2667 trials with 8 times of success, then the successful rate is 0.003 = 8 / 2667. However, I can also do 26667 with 80 times of success, the successful rate is also 0.003 = 80 / 26667. In the same case, the successful rate can also from this: 0.003 = 800 / 266667. In such a case, which number of trials (2667, 26667 or 266667) should be selected? From economic perspective, I will select 2667 because it save money and time, but from a statistical point of view, is 2667 enough to get a statistically important result (proportion of 0.003)? I hope this time it is clear. Please correct me if I am still wrong. Thanks.

Please ignore the following inputs which was considered as unclear, I edited my question above.

Currently, I faced such a question for doing the statistical testing based on the proportion (testing if a new idea can improve the successful rate). In the control group, I did 26,667 trials and the successful times were 80, then the proportion is about 0.003; in the treatment group, I did the same number of 26,667 trials, the successful time were also 80 which means the treatment group also gave the same proportion 0.003, indicating the new idea is not helpful for successful rate. The question is: if in the control and treatment groups, the proportions of success are both 0.003, then what is the minimal sample size for statistical testing of the two equal proportions? I understand that if I want to determine the sample size for testing two different proportions given the significant level and power in R. for example:

power.prop.test(p1 = 0.003, p2 = 0.004, sig.level = 0.05, power = 0.8)

then the sample size is about 54748 for each group of control and treatment. However, my question is if the two proportions in control and treatment groups are equal. In such a case, what is the minimal sample size for testing the equal proportions? For example, instead of doing 26,667 trial with 80 success results, I can do 2667 with 8 success results, then for both cases the successful rates are 0.003. So I just need 2,667 trials rather than 26,667 to get the result. How do I determine the sample size 2,667 versus 26,667 for testing the equal proportions statistically? or my question does not make sense at all? Basically, I just asked the opposite question compared with ordinary question (i.e., given two equal proportions rather than different proportions, how to determine the minimal sample size by fixing power and significant level).

Thank you for giving useful suggestions in the reply.

To clear the question, I can ask like this. If I want to get the successful rate is 0.003, how many trials (each trial has one of success or failure results) are needed, making such a proportion (0.003) is statistically important? In theory, I can do 100 trials, 1000 trials, even more trials like 10,000 trials (however, from the practical perspective, I should use the minimal trials because it saves money and time, but result is still statistically important). So the question is what is the minimal trials are needed to get the statistically important successful rate of 0.003. Here I used the power of 0.8, and significant level of 0.05. Please help. Correct me if my question makes no sense. Thanks.

Based on the input from two reply. I got the idea, that is to determine the sample size, two conditions must be given: (1) the effect/difference (e.g. 0.001 = p2 - p1) and the power (e.g. beta = 0.8), then at the significant level of alpha = 0.05, sample size can be estimated using the function like the one I posted above. If no this two conditions, sample size can not be estimated in my case.

Thank you.

$\endgroup$
  • 3
    $\begingroup$ Your question doesn't make any sense to me. It requires no sample size at all to fail to reject the null, and you certainly can't do anything like prove a point null in that situation at any sample size. It's not even clear to me whether you have misunderstood power or whether you are seeking something nearer to an equivalence test. Perhaps you should start by explaining the underlying question of interest you're trying to answer from data. $\endgroup$ – Glen_b -Reinstate Monica Sep 2 at 3:56
  • $\begingroup$ @Glen_b thanks. I edited it hoping make it more clear. $\endgroup$ – RMathStatR Sep 2 at 13:02
  • $\begingroup$ It still asks " what is the minimal sample size for statistical testing of the two equal proportions" which is the part that doesn't make sense to me. The fact that you observe two equal sample proportions has nothing to do with working out sample (you already have the samples those proportions come from; you know how big they were). Can you explain why you're using sample information in a power calculation? $\endgroup$ – Glen_b -Reinstate Monica Sep 2 at 15:29
  • $\begingroup$ @Glen_b Please have a look at the new edited one. Thanks. $\endgroup$ – RMathStatR Sep 3 at 1:44
  • $\begingroup$ It's still not very clear to me what you are trying to do here. 1. Your original question seemed to relate to comparing two proportions. Now you only seem to be asking about a single proportion. Is that right? 2. Are you sure a hypothesis test is answering your question of interest? 3. Power is a property that applies when the null hypothesis is false, not when it is true (It's like saying "what's the chance of catching a guilty person when they're innocent?). You can't work out power for a true null. .... $\endgroup$ – Glen_b -Reinstate Monica Sep 3 at 2:49
4
$\begingroup$

if in the control and treatment groups, the proportions of success are both 0.003, then what is the minimal sample size for statistical testing of the two equal proportions

When you are doing hypothesis testing then the null hypothesis, when it is true, will be rejected by the significance level $\alpha$ that you choose, or when the null hypothesis is not true, it will be rejected by a rate that is ideally much higher than the significance level.

What is important is not only the case "the proportions of success are both 0.003", but instead also the cases when those proportions are different. The more different the proportions are, the more probable it becomes that you will observe a significant difference and reject the null hypothesis.

In order to determine what size of sample is neccesary to take, you could express something like the probability to observe a significant difference, given a true difference (of some specific effect size), as function of the sample sizes. So to compute the sample size you need 1) an idea of a relevant minimal difference/effect 2) a level of desired power/probability.

It is important to specify this minimal difference, since in practice the null hypothesis is almost never true. Some way or another the different treatment might have a tiny miniscule effect (not of the kind of size that was theoretically expected) and given a large enough sample you might show that the two groups are different by a tiny minuscule amount.

When doing hypothesis testing, we often challenge the null hypothesis (there is no effect) in order to show whether there is an effect or not. But what researchers might actually be interested in is to challenge the alternative hypothesis (there is an effect) in order to show whether the hypothesized effect is true or not.

Note: There is a difference between 'not rejecting the null hypothesis' and 'rejecting the alternative hypothesis'.

Two ways to deal with this type of problem are two one-sided t-tests (TOST) and likelihood ratio test. In both cases you explicitly specify both the hypotheses (null/alternative).


To the point: To do the computations of sample size you can approximate the variables as normal distributed. In a simple way you use the 0.003 as an initial value by which you can compute the variance, but a more difficult case is when the proportions turn out to be smaller than initially expected (which reduces the number of successes and you actually wish to have a certain number of successes rather than a certain number of total sample).

$\endgroup$
  • $\begingroup$ thank you. I re-edited it to make it more clear. My question actually is that what is the minimal sample size to get a statistically important successful rate (i.e., 0.003, given the power of 0.8 and significant level of 0.05). I am not sure if the question make any sense. Please help. $\endgroup$ – RMathStatR Sep 2 at 13:05
  • $\begingroup$ @RMathStatR your edit is not really sufficient. Note this comment in my answer: "So to compute the sample size you need 1) an idea of a relevant minimal difference/effect 2) a level of desired power/probability. " The power is dependent on the effect size (see also the curves made by BruceET). If you only specify that you desire a power of 0.8, then you are not complete. You should also specify for which effect size this power is desired. $\endgroup$ – Sextus Empiricus Sep 2 at 13:13
  • $\begingroup$ thank you. Got your suggestions. May I ask one more question? If I just want to determine the initial successful rate (e.g. 0.003), how many trials are needed from the statistical point of view? $\endgroup$ – RMathStatR Sep 2 at 14:09
  • 1
    $\begingroup$ @RMathStatR, it depends on the precision by which you wish to estimate the successful rate. You could approximate the number of successes (a binomial distributed variable) as a normal distributed variable and with variance and mean equal to $\mu=np$ and $\sigma^2 = np(1-p) \approx np$. The estimate of the succesful rate will be $\hat{p} = \text{observed proportion}$ with an estimated standard error of approximately $\frac{\hat{p}}{\sqrt{n}}$. $\endgroup$ – Sextus Empiricus Sep 2 at 14:25
  • $\begingroup$ thank you. If I just use the binomial distributed variable itself, rather than a approximated normal distributed variable, is there a way to estimated the initial successful rate? $\endgroup$ – RMathStatR Sep 3 at 1:20
3
$\begingroup$

@MartijnWitherings (+1) has given a nice orientation and introduction to this topic. For reference, here are a couple of specific power computations.

Testing at the 5% level, suppose you want to know sample sizes for two groups that will give a probability (power) .90 [or .95] of detecting a true difference in proportions of 0.003 and 0.01. Then Minitab's power and sample size procedure says you should use about 2768 [or 3423] subjects in each group.

 Power and Sample Size 

 Test for Two Proportions

 Testing comparison p = baseline p (versus ≠)
 Calculating power for baseline p = 0.003
 α = 0.05


               Sample  Target
 Comparison p    Size   Power  Actual Power
         0.01    2768    0.90      0.900053
         0.01    3423    0.95      0.950043

 The sample size is for each group.

enter image description here

From the following, you can see that much larger sample sizes are necessary to detect a difference between 0.003 and 0.005. In planning sample size for an experiment, it is important to know the size of the difference that is of practical importance or scientific interest.

 Power and Sample Size 

 Test for Two Proportions

 Testing comparison p = baseline p (versus ≠)
 Calculating power for baseline p = 0.003
 α = 0.05


               Sample  Target
 Comparison p    Size   Power  Actual Power
        0.005   20929    0.90      0.900004
        0.005   25883    0.95      0.950004

  The sample size is for each group.

enter image description here

$\endgroup$
  • $\begingroup$ thank you. It is very helpful. I re-edited the question. Basically, if I just for the first time do an example, and I want to get the successful rate of 0.003, given the power of 0.8 and significant level of 0.05, then what is the minimal trials I need to get the rate of 0.003 statistically important? $\endgroup$ – RMathStatR Sep 2 at 13:08
  • $\begingroup$ @ RMathStatR. Puzzled by your question. Chances of getting exactly 0.003 are tiny. Do you mean within some margin of error of 0.003? Do you mean at least 0.003? (And how does 0.003 arise in this context? Is that a value precisely predicted by theory? The value you got the last time you did the experiment?) $\endgroup$ – BruceET Sep 2 at 17:52
  • 1
    $\begingroup$ 0.003 here is just a fake value, but in the real world, some cases such as response rate may get the response rate of 0.003. According to Martijn Weterings and your explanation, I now understand that in order to get sample size, I need specify the difference/effect and power ahead of time. $\endgroup$ – RMathStatR Sep 2 at 22:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.