# Significance level in sample size calculation and in final analysis

Imagine that for the purpose of a study a sample size is computed with the following formula for a given power $1-\beta$, difference in means $\epsilon$, standard deviation $\sigma$ and significance level $\alpha$ $$n = \frac{2(z_{\alpha/2} + z _{\beta})\sigma^2}{\epsilon^2}$$

When all $n$ subjects completed the study, we have the data required to perform the analysis, e.g. t-test. When I apply this t-test, does the significance level $\alpha$ has to be the same as the one used in the sample size calculation?

Are the choices of $\alpha$ "before" and "after" the conduct of the study, if may say so, have to be identical?

No they don't "have" to be the same. The $\alpha$ you used in your power analysis before conducting the study, and the $\alpha$ you used in your test don't technically have to be the same. But, if you're only "relaxing" $\alpha$ in your test because you didn't get the result you wanted/expected -- and you're only changing this between your design and your analysis because you want to report a "statistically significant" result -- I would discourage this. That's not how hypothesis testing works. Instead, report your design; report your results (even if $p \ge 0.05$); and discuss the implications. Don't let your p-value be your only metric of success or failure in your research. Null findings can also have a great deal of scientific merit.
• Thank you for the reply. But what I had in mind concerned co-primary endpoints. So when you say they don't "have" to be the same, does it make sense to compute the sample size with an $\alpha=0.025$ and then perform the t-test with an $\alpha=0.05$ given that this strategy was planned from the begining? This means we were conservative while computing the sample size. Wasn't this enough? Do we still have to keep a $\alpha=0.025$ for the t-test? May 30, 2016 at 17:22
• I mean what would "technically" happen in terms of interpretation if the $\alpha$ changes? From a small $\alpha$ to a bigger one and the opposite? May 30, 2016 at 17:28
• You already have your sample, so with whatever $\alpha$ you choose for your test, $n$ isn't going to change in your calculation of a t-statistic. Accepting a larger $\alpha$ means that you're willing to accept a greater likelihood of a false positive, and that your standard errors will be larger, but there are far better discussions of $\alpha$ and p-values on this site than can be summarized in a comment. Point being that you can use a different $\alpha$ in your test if you have a good reason, but in your write up, you should flag any deviations from your initial hypothesis/design. May 30, 2016 at 20:47
The $\alpha$ is usually 0.05 unless multiple tests correction is applied.
(The problem is usually in the effect size (in other words, in $\epsilon$). If the effect size is not exactly known, the sample size calculations are only approximate. Even after the experiment is done you can't know the power.)