0
$\begingroup$

Let's assume that the researcher assumes an event probability of 70% and 60% in the study and control groups, respectively for the sample size calculation.

However, at the end of the study, the event occurred in only 40% and 30% of the study and control groups, respectively. Despite this, the P-value is statistically significant (P=0.01). The researcher concludes the study on a positive note.

Is the result of this study valid?

Should this study be called underpowered?

EDIT: A more detailed explanation of the question is as follows: A disease has a mortality rate of 70% despite the available standard treatment. Researchers wish to evaluate a new drug for this disease. The researchers hypothesize that the new drug will reduce the mortality to 60%. They design a clinical trial to compare the new drug (study arm) and the standard treatment (control arm). They arrive at a sample size of 476 in each arm for a power of 90% and a two-sided significance level of 5% (using pwr.2p.test in R).

However, for some unknown reason, the actual mortality observed in the study was lower than expected (55% in the control arm and 50% in the study arm). But, the researchers obtained a statistically significant P-value (0.01) for this difference. They concluded that the new drug is better than the standard treatment.

Is this conclusion valid?

The question arises because of the unexpected change in the effect size and the consequent change needed in the sample size. The effect size (Cohen's h) for a mortality rate of 70% vs. 60% is 0.21. However, the effect size for a mortality rate of 55% vs. 50% is only 0.1. Because of the reduction in the effect size, the sample size needed would be much higher (2095 in each arm).

If the study had used the originally planned sample size of 476 in each arm, shouldn't it be called as underpowered?

Additionally, I believe that obtaining a P-value of 0.01 using the original sample size (476 in each arm) is not possible without the introduction of some form of error during the study. I came to this conclusion after the two-proportions z-test in R (I believe it is the same as the chi-squared test. I'm just a novice in R!) gave P of 0.001771 and 0.1355 for the comparison of 60% vs. 70% and 50% vs. 55%, respectively.

PS: The effect size for the mortality rate of 40% vs. 30% in the original question is also 0.21. Hence, I have altered it.

$\endgroup$
2
  • $\begingroup$ More context is needed for a truly responsive "Answer". I speculate below. $\endgroup$
    – BruceET
    Sep 17, 2021 at 17:44
  • $\begingroup$ I have edited the question to include a more detailed explanation. $\endgroup$
    – Bradex
    Sep 20, 2021 at 4:19

1 Answer 1

1
$\begingroup$

If this is a question about an actual research project, then it would be appropriate to say what sample size is being used, to state the hypotheses, to say what kind of "events" are under study, and to explain the objectives of the study.

Also, the phrase "at the end of the study" is difficult to interpret. Does it mean that the proportion of occurrences changed from 70%/60% at the start of the study to 30%/40% and the end? Or does it mean that initial assumptions about the proportion of "events" (for the sample size computation) turned out to be surprisingly wrong.

However, I suspect this is a somewhat awkwardly worded textbook problem and you are supposed to say the two situations are equivalent. Just focus on occurrence of the "events" (Successes) in the first part and on their non-occurrence (Failures) in the second.

Below are analyses of two scenarios with $n = 500$ and exactly the results you describe, using the procedure prop.test in R. P-values are identical.

[The distributions $\mathsf{Binom}(n, .7)$ and $\mathsf{Binom}(n, .3)$ have the same variance, as do $\mathsf{Binom}(n, .6)$ and $\mathsf{Binom}(n, .4).$ However, $\mathsf{Binom}(n, .5)$ has a little larger variance.]

n = 500
prop.test(c(.3*n,.4*n), c(n,n))

        2-sample test for equality of proportions 
        with continuity correction

data:  c(0.3 * n, 0.4 * n) out of c(n, n)
X-squared = 10.554, df = 1, p-value = 0.001159
alternative hypothesis: two.sided
95 percent confidence interval:
 -0.16079892 -0.03920108
sample estimates:
prop 1 prop 2 
   0.3    0.4 


prop.test(c(.6*n,.7*n), c(n,n))

        2-sample test for equality of proportions 
        with continuity correction

data:  c(0.6 * n, 0.7 * n) out of c(n, n)
X-squared = 10.554, df = 1, p-value = 0.001159
alternative hypothesis: two.sided
95 percent confidence interval:
 -0.16079892 -0.03920108
sample estimates:
prop 1 prop 2 
   0.6    0.7 
$\endgroup$
3
  • $\begingroup$ Cohen's $d$ is problematic in general and should not be used for binary variables IMHO. Power is a pre-study issue and should not come into play at this point. Other than those considerations, if Y is truly binary (and not time to event) then the question comes down to accuracy of calculations, noting that in the frequentist world there is no exact unconditional test for this situation. $\endgroup$ Sep 18, 2021 at 11:58
  • $\begingroup$ I have used Cohen's h and not Cohen's d. $\endgroup$
    – Bradex
    Sep 18, 2021 at 13:10
  • $\begingroup$ I agree that power is a pre-study issue. However, if the pre-study assumptions are not met in the actual study, can it be still called adequately powered? If whether or not the study live up to these pre-study assumptions does not matter, then why bother to make such assumptions at all? Further, anybody can calculate a 'desired' sample size by manipulating the effect size. At the end of the study, when they experience a much lower effect size, they can always say that it is only a pre-study issue. $\endgroup$
    – Bradex
    Sep 20, 2021 at 3:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.