# How to interpret P-value result in light of P-value used for sample size calculation

A study is planned comparing effects of 2 drugs. Sample size is calculated for power of 0.8 and alpha of 0.01

After study is completed, the results show that there is a difference between 2 groups and P-value is 0.03

How does one interpret this result?

Do we take it as significant since it is <0.05, the conventional cut-off?

Or we say that this result is not significant since it is >0.01 (alpha used for sample size calculation)?

In general, how do we analyze study results in view of P-value taken for sample size calculations?

Edit: Hypothesis is that one of these drugs is better than other. Measure of effectiveness is a continuous variable like blood levels of a chemical.

I suspected same value should be used after results are obtained but some discussion here seemed to indicate that the two are different.

• (1) You don't say what measure of effectiveness of the drugs you are using. (2) If this is a clinical trial and the sample sizes were stated in a government-approved protocol for signif level 1%, then your results fail the official objective. (Perhaps variability of responses to one or both drugs was underestimated in planning.) (3) To decide whether to abandon study of a new drug, it might be worthwhile to assess whether the observed difference between the drugs (if real) would be of practical importance. – BruceET Jul 1 at 20:52
• This reads like a homework problem, but I'm guessing this is a contrived example to help understand a nuance of hypothesis testing. In case this was real, what was the hypothesis here? Specifically, was this a non-inferiority trial? – gung - Reinstate Monica Jul 1 at 21:10
• My other questions and reputation here and at stackoverflow.com/users/3522130/rnso could confirm that this is not a homework problem. It is not a real situation and you are right that I am trying to understand a nuance of hypothesis testing. The hypothesis is that one of these drugs is better than other. – rnso Jul 2 at 1:33
• @BruceET Measure of effectiveness is a continuous variable like blood levels of a chemical. I think you can put the points in your comment as a full answer. I wanted to confirm that in this situation, as you say, results fail the official objective. Will the answer be different if measure of improvement was cateogorical, such as proportion of patients surviving illness? – rnso Jul 2 at 1:40
• OK. I had wondered whether it might be useful to discuss this using simulated data. I have not deleted any of my Comments, in case they may still be useful. But I have put parts of the discussion into a formal Answer as you asked. – BruceET Jul 2 at 5:23

Suppose your running a clinical trial to see whether new Drug 2 is better than current Drug 1. Drug effectiveness will be judged as increased blood levels of a particular enzyme.

With a 'power and sample size' procedure you have determined that $$n_1 = n_2 = 1000$$ subjects in treatment and control groups will suffice to find an increase enzyme levels by a clinically useful amount (if there really is such a difference) at the 1% level of statistical significance. The protocol for this trial has been approved.

Because enzyme levels are approximately normally distributed for subjects similar to those in this trial, it is agreed that a two-sample Welch t test will be used for the primary statistical analysis.

Data are found to be as in vectors x1 and x2 summarized and described as below:

summary(x1); sd(x1)
Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
310.0   457.2   496.5   498.4   538.3   722.2
[1] 62.21792  # SD of x1
summary(x2); sd(x2)
Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
295.9   466.6   501.1   504.1   544.5   722.9
[1] 60.36752

hdr="Enzyme Levels"
boxplot(x1, x2, col="skyblue2", pch=20, names=T, main=hdr)
abline(h=500, col="green2")
points(1:2, c(mean(x1),mean(x2)), pch="X", col="red")


Cross bars in the boxplots show median enzyme levels, red xs show mean enzyme levels. The horizontal green reference line is at 500. It is not unusual to for normal samples of size 1000 to show some outliers (heavy dots), and they can be ignored in the analysis.

A Welch two-sample t test shows that the new drug had average enzyme levels 5.7 units greater than for the current drug, but this difference is too small to be statistically significant at the 1% level (even though it is significant at the 4% level).

t.test(x1, x2)

Welch Two Sample t-test

data:  x1 and x2
t = -2.0805, df = 1996.2, p-value = 0.0376
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-11.0799116  -0.3272779
sample estimates:
mean of x mean of y
498.3993  504.1029


Thus the trial failed its objectives according to the government-approved protocol. Several factors might have entered into the failure. The standard deviations of both drugs were larger in the trial, and the performance of the new drug was a little smaller than expected.

The question remains whether it is worthwhile to continue with development and testing of the new drug. The new drug may have some advantages in simplicity of manufacture, fewer side-effects. And there is some evidence that it may be a little better than the existing drug at raising enzyme levels. And earlier trials indicated it has sometimes performed better than in the current trial. Earlier trials also suggested that higher doses might be feasible without unwelcome side effects.

So it may be worthwhile investigating whether the new drug might be a viable competitor to the existing one. However, further development of the new drug will require another clinical trial with a new protocol.

Note: If the measure of usefulness of a new drug is based on patient and physician questionnaires about perceived improvement similar issues might arise. Nonparametric tests on questionnaire results might also show significance at the 5% level, when protocols required the 1% level. Power and sample size procedures may be more difficult for nonparametric tests.

Note: The data for the test above were simulated in R as follows:

set.seed(2020)
x1 = rnorm(1000, 500, 60)
x2 = rnorm(1000, 504, 60)
t.test(x1,x2)$p.val [1] 0.03760362  • Thanks for a detailed answer. At stats.stackexchange.com/questions/33453/… it seems that alpha and P-value of outcome are different? – rnso Jul 2 at 5:27 • Different situations, different data. Puzzled why you expect them to be the same. – BruceET Jul 2 at 5:39 • Ok. Good point. I will see it again from that point. – rnso Jul 2 at 6:35 • Perhaps worth adding that if the protocol is published any high-quality medical journal will check this and raise issues if the$\alpha\$ has been changed. One might also add though that the study should still be published but just with the caveat that it failed to reach the arbitrary level set out. – mdewey Jul 2 at 12:19

As in all research, you should specify your test and your level of statistical significance prior to analyzing your data.

So if you used $$\alpha = 0.01$$ to figure out sample size calculations, then that's presumably the level you choose for your study, and so you should compare it to that. Otherwise why was that the $$\alpha$$ you choose for sample size calculations? It should also be emphasized that the 'conventional' cut-off of $$.05$$ is only true in some fields, and regardless always must be chosen before analyzing your data. In some fields, $$\alpha = .01$$ is the convention, and in others, $$\alpha = .1$$ is the convention. By altering your $$\alpha$$ after observing data, what you would be doing is no different than choosing confidence level greater than your pvalue after observing your pvalue, which is completely counter to how it should be done.