If the observed event probability is lower than the assumed event probability used for sample size calculation will the trial become underpowered?

Question

Let's assume that the researcher assumes an event probability of 70% and 60% in the study and control groups, respectively for the sample size calculation.

However, at the end of the study, the event occurred in only 40% and 30% of the study and control groups, respectively. Despite this, the P-value is statistically significant (P=0.01). The researcher concludes the study on a positive note.

Is the result of this study valid?

Should this study be called underpowered?

EDIT: A more detailed explanation of the question is as follows: A disease has a mortality rate of 70% despite the available standard treatment. Researchers wish to evaluate a new drug for this disease. The researchers hypothesize that the new drug will reduce the mortality to 60%. They design a clinical trial to compare the new drug (study arm) and the standard treatment (control arm). They arrive at a sample size of 476 in each arm for a power of 90% and a two-sided significance level of 5% (using pwr.2p.test in R).

However, for some unknown reason, the actual mortality observed in the study was lower than expected (55% in the control arm and 50% in the study arm). But, the researchers obtained a statistically significant P-value (0.01) for this difference. They concluded that the new drug is better than the standard treatment.

Is this conclusion valid?

The question arises because of the unexpected change in the effect size and the consequent change needed in the sample size. The effect size (Cohen's h) for a mortality rate of 70% vs. 60% is 0.21. However, the effect size for a mortality rate of 55% vs. 50% is only 0.1. Because of the reduction in the effect size, the sample size needed would be much higher (2095 in each arm).

If the study had used the originally planned sample size of 476 in each arm, shouldn't it be called as underpowered?

Additionally, I believe that obtaining a P-value of 0.01 using the original sample size (476 in each arm) is not possible without the introduction of some form of error during the study. I came to this conclusion after the two-proportions z-test in R (I believe it is the same as the chi-squared test. I'm just a novice in R!) gave P of 0.001771 and 0.1355 for the comparison of 60% vs. 70% and 50% vs. 55%, respectively.

PS: The effect size for the mortality rate of 40% vs. 30% in the original question is also 0.21. Hence, I have altered it.

Answer 1

If this is a question about an actual research project, then it would be appropriate to say what sample size is being used, to state the hypotheses, to say what kind of "events" are under study, and to explain the objectives of the study.

Also, the phrase "at the end of the study" is difficult to interpret. Does it mean that the proportion of occurrences changed from 70%/60% at the start of the study to 30%/40% and the end? Or does it mean that initial assumptions about the proportion of "events" (for the sample size computation) turned out to be surprisingly wrong.

However, I suspect this is a somewhat awkwardly worded textbook problem and you are supposed to say the two situations are equivalent. Just focus on occurrence of the "events" (Successes) in the first part and on their non-occurrence (Failures) in the second.

Below are analyses of two scenarios with $n = 500$ and exactly the results you describe, using the procedure prop.test in R. P-values are identical.

[The distributions $\mathsf{Binom}(n, .7)$ and $\mathsf{Binom}(n, .3)$ have the same variance, as do $\mathsf{Binom}(n, .6)$ and $\mathsf{Binom}(n, .4).$ However, $\mathsf{Binom}(n, .5)$ has a little larger variance.]

n = 500
prop.test(c(.3*n,.4*n), c(n,n))

        2-sample test for equality of proportions 
        with continuity correction

data:  c(0.3 * n, 0.4 * n) out of c(n, n)
X-squared = 10.554, df = 1, p-value = 0.001159
alternative hypothesis: two.sided
95 percent confidence interval:
 -0.16079892 -0.03920108
sample estimates:
prop 1 prop 2 
   0.3    0.4 


prop.test(c(.6*n,.7*n), c(n,n))

        2-sample test for equality of proportions 
        with continuity correction

data:  c(0.6 * n, 0.7 * n) out of c(n, n)
X-squared = 10.554, df = 1, p-value = 0.001159
alternative hypothesis: two.sided
95 percent confidence interval:
 -0.16079892 -0.03920108
sample estimates:
prop 1 prop 2 
   0.6    0.7