Let's assume that the researcher assumes an event probability of 70% and 60% in the study and control groups, respectively for the sample size calculation.
However, at the end of the study, the event occurred in only 40% and 30% of the study and control groups, respectively. Despite this, the P-value is statistically significant (P=0.01). The researcher concludes the study on a positive note.
Is the result of this study valid?
Should this study be called underpowered?
EDIT: A more detailed explanation of the question is as follows: A disease has a mortality rate of 70% despite the available standard treatment. Researchers wish to evaluate a new drug for this disease. The researchers hypothesize that the new drug will reduce the mortality to 60%. They design a clinical trial to compare the new drug (study arm) and the standard treatment (control arm). They arrive at a sample size of 476 in each arm for a power of 90% and a two-sided significance level of 5% (using pwr.2p.test in R).
However, for some unknown reason, the actual mortality observed in the study was lower than expected (55% in the control arm and 50% in the study arm). But, the researchers obtained a statistically significant P-value (0.01) for this difference. They concluded that the new drug is better than the standard treatment.
Is this conclusion valid?
The question arises because of the unexpected change in the effect size and the consequent change needed in the sample size. The effect size (Cohen's h) for a mortality rate of 70% vs. 60% is 0.21. However, the effect size for a mortality rate of 55% vs. 50% is only 0.1. Because of the reduction in the effect size, the sample size needed would be much higher (2095 in each arm).
If the study had used the originally planned sample size of 476 in each arm, shouldn't it be called as underpowered?
Additionally, I believe that obtaining a P-value of 0.01 using the original sample size (476 in each arm) is not possible without the introduction of some form of error during the study. I came to this conclusion after the two-proportions z-test in R (I believe it is the same as the chi-squared test. I'm just a novice in R!) gave P of 0.001771 and 0.1355 for the comparison of 60% vs. 70% and 50% vs. 55%, respectively.
PS: The effect size for the mortality rate of 40% vs. 30% in the original question is also 0.21. Hence, I have altered it.