How to interpret P-value result in light of P-value used for sample size calculation A study is planned comparing effects of 2 drugs. Sample size is calculated for power of 0.8 and alpha of 0.01
After study is completed, the results show that there is a difference between 2 groups and P-value is 0.03
How does one interpret this result?
Do we take it as significant since it is <0.05, the conventional cut-off?
Or we say that this result is not significant since it is >0.01 (alpha used for sample size calculation)?
In general, how do we analyze study results in view of P-value taken for sample size calculations?
Edit: Hypothesis is that one of these drugs is better than other. Measure of effectiveness is a continuous variable like blood levels of a chemical.
I suspected same value should be used after results are obtained but some discussion here seemed to indicate that the two are different.
 A: As in all research, you should specify your test and your level of statistical significance prior to analyzing your data.
So if you used $\alpha = 0.01$ to figure out sample size calculations, then that's presumably the level you choose for your study, and so you should compare it to that. Otherwise why was that the $\alpha$ you choose for sample size calculations? It should also be emphasized that the 'conventional' cut-off of $.05$ is only true in some fields, and regardless always must be chosen before analyzing your data. In some fields, $\alpha = .01$ is the convention, and in others, $\alpha = .1$ is the convention. By altering your $\alpha$ after observing data, what you would be doing is no different than choosing confidence level greater than your pvalue after observing your pvalue, which is completely counter to how it should be done.
A: Suppose your running a clinical trial to see whether new Drug 2
is better than current Drug 1. Drug effectiveness will be judged
as increased blood levels of a particular enzyme.
With a 'power
and sample size' procedure you have determined that $n_1 = n_2 = 1000$
subjects in treatment and control groups will suffice to find
an increase enzyme levels by a clinically useful amount (if there
really is such a difference) at the 1% level of statistical
significance. The protocol for this trial has been approved.
Because enzyme levels are approximately normally distributed
for subjects similar to those in this trial, it is agreed that a two-sample Welch t test will be used for the primary statistical analysis.
Data are found to be as in vectors x1 and x2 summarized and
described as below:
summary(x1); sd(x1)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  310.0   457.2   496.5   498.4   538.3   722.2 
[1] 62.21792  # SD of x1
summary(x2); sd(x2)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  295.9   466.6   501.1   504.1   544.5   722.9 
[1] 60.36752

hdr="Enzyme Levels"
boxplot(x1, x2, col="skyblue2", pch=20, names=T, main=hdr)
 abline(h=500, col="green2")
 points(1:2, c(mean(x1),mean(x2)), pch="X", col="red")


Cross bars in the boxplots show median enzyme levels, red xs show
mean enzyme levels. The horizontal green reference line is at 500.
It is not unusual to for normal samples of size 1000 to show some
outliers (heavy dots), and they can be ignored in the analysis.
A Welch two-sample t test shows that the new drug had average enzyme
levels 5.7 units greater than for the current drug, but this difference
is too small to be statistically significant at the 1% level (even
though it is significant at the 4% level).
t.test(x1, x2)

        Welch Two Sample t-test

data:  x1 and x2
t = -2.0805, df = 1996.2, p-value = 0.0376
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -11.0799116  -0.3272779
sample estimates:
mean of x mean of y 
 498.3993  504.1029 

Thus the trial failed its objectives according to the government-approved protocol.
Several factors might have entered into the failure. The standard deviations of
both drugs were larger in the trial, and the performance of the new drug was
a little smaller than expected.
The question remains whether it is worthwhile to continue with development and
testing of the new drug.  The new drug may have some advantages in simplicity
of manufacture, fewer side-effects. And there is some evidence that it may
be a little better than the existing drug at raising enzyme levels. And earlier
trials indicated it has sometimes performed better than in the current trial.
Earlier trials also suggested that higher doses might be feasible without
unwelcome side effects.
So it may be worthwhile investigating whether the new drug might be a viable
competitor to the existing one.
However, further development of the new drug will require another clinical
trial with a new protocol.
Note: If the measure of usefulness of a new drug is based on patient and
physician questionnaires about perceived improvement similar issues might arise.
Nonparametric tests on questionnaire results might also show significance
at the 5% level, when protocols required the 1% level. Power and sample size
procedures may be more difficult for nonparametric tests.
Note: The data for the test above were simulated in R as follows:
set.seed(2020)
x1 = rnorm(1000, 500, 60)
x2 = rnorm(1000, 504, 60)
t.test(x1,x2)$p.val
[1] 0.03760362

