How to interpret P-value result in light of P-value used for sample size calculation

Question

A study is planned comparing effects of 2 drugs. Sample size is calculated for power of 0.8 and alpha of 0.01

After study is completed, the results show that there is a difference between 2 groups and P-value is 0.03

How does one interpret this result?

Do we take it as significant since it is <0.05, the conventional cut-off?

Or we say that this result is not significant since it is >0.01 (alpha used for sample size calculation)?

In general, how do we analyze study results in view of P-value taken for sample size calculations?

Edit: Hypothesis is that one of these drugs is better than other. Measure of effectiveness is a continuous variable like blood levels of a chemical.

I suspected same value should be used after results are obtained but some discussion here seemed to indicate that the two are different.

Answer 1

Suppose your running a clinical trial to see whether new Drug 2 is better than current Drug 1. Drug effectiveness will be judged as increased blood levels of a particular enzyme.

With a 'power and sample size' procedure you have determined that $n_1 = n_2 = 1000$ subjects in treatment and control groups will suffice to find an increase enzyme levels by a clinically useful amount (if there really is such a difference) at the 1% level of statistical significance. The protocol for this trial has been approved.

Because enzyme levels are approximately normally distributed for subjects similar to those in this trial, it is agreed that a two-sample Welch t test will be used for the primary statistical analysis.

Data are found to be as in vectors x1 and x2 summarized and described as below:

summary(x1); sd(x1)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  310.0   457.2   496.5   498.4   538.3   722.2 
[1] 62.21792  # SD of x1
summary(x2); sd(x2)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  295.9   466.6   501.1   504.1   544.5   722.9 
[1] 60.36752

hdr="Enzyme Levels"
boxplot(x1, x2, col="skyblue2", pch=20, names=T, main=hdr)
 abline(h=500, col="green2")
 points(1:2, c(mean(x1),mean(x2)), pch="X", col="red")

Cross bars in the boxplots show median enzyme levels, red xs show mean enzyme levels. The horizontal green reference line is at 500. It is not unusual to for normal samples of size 1000 to show some outliers (heavy dots), and they can be ignored in the analysis.

A Welch two-sample t test shows that the new drug had average enzyme levels 5.7 units greater than for the current drug, but this difference is too small to be statistically significant at the 1% level (even though it is significant at the 4% level).

t.test(x1, x2)

        Welch Two Sample t-test

data:  x1 and x2
t = -2.0805, df = 1996.2, p-value = 0.0376
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -11.0799116  -0.3272779
sample estimates:
mean of x mean of y 
 498.3993  504.1029 

Thus the trial failed its objectives according to the government-approved protocol. Several factors might have entered into the failure. The standard deviations of both drugs were larger in the trial, and the performance of the new drug was a little smaller than expected.

The question remains whether it is worthwhile to continue with development and testing of the new drug. The new drug may have some advantages in simplicity of manufacture, fewer side-effects. And there is some evidence that it may be a little better than the existing drug at raising enzyme levels. And earlier trials indicated it has sometimes performed better than in the current trial. Earlier trials also suggested that higher doses might be feasible without unwelcome side effects.

So it may be worthwhile investigating whether the new drug might be a viable competitor to the existing one. However, further development of the new drug will require another clinical trial with a new protocol.

Note: If the measure of usefulness of a new drug is based on patient and physician questionnaires about perceived improvement similar issues might arise. Nonparametric tests on questionnaire results might also show significance at the 5% level, when protocols required the 1% level. Power and sample size procedures may be more difficult for nonparametric tests.

Note: The data for the test above were simulated in R as follows:

set.seed(2020)
x1 = rnorm(1000, 500, 60)
x2 = rnorm(1000, 504, 60)
t.test(x1,x2)$p.val
[1] 0.03760362

Answer 2

As in all research, you should specify your test and your level of statistical significance prior to analyzing your data.

So if you used $\alpha = 0.01$ to figure out sample size calculations, then that's presumably the level you choose for your study, and so you should compare it to that. Otherwise why was that the $\alpha$ you choose for sample size calculations? It should also be emphasized that the 'conventional' cut-off of $.05$ is only true in some fields, and regardless always must be chosen before analyzing your data. In some fields, $\alpha = .01$ is the convention, and in others, $\alpha = .1$ is the convention. By altering your $\alpha$ after observing data, what you would be doing is no different than choosing confidence level greater than your pvalue after observing your pvalue, which is completely counter to how it should be done.