1
$\begingroup$

Imagine that for the purpose of a study a sample size is computed with the following formula for a given power $1-\beta$, difference in means $\epsilon$, standard deviation $\sigma$ and significance level $\alpha$ \begin{equation} n = \frac{2(z_{\alpha/2} + z _{\beta})\sigma^2}{\epsilon^2} \end{equation}

When all $n$ subjects completed the study, we have the data required to perform the analysis, e.g. t-test. When I apply this t-test, does the significance level $\alpha$ has to be the same as the one used in the sample size calculation?

Are the choices of $\alpha$ "before" and "after" the conduct of the study, if may say so, have to be identical?

$\endgroup$
2
$\begingroup$

No they don't "have" to be the same. The $\alpha$ you used in your power analysis before conducting the study, and the $\alpha$ you used in your test don't technically have to be the same. But, if you're only "relaxing" $\alpha$ in your test because you didn't get the result you wanted/expected -- and you're only changing this between your design and your analysis because you want to report a "statistically significant" result -- I would discourage this. That's not how hypothesis testing works. Instead, report your design; report your results (even if $p \ge 0.05$); and discuss the implications. Don't let your p-value be your only metric of success or failure in your research. Null findings can also have a great deal of scientific merit.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you for the reply. But what I had in mind concerned co-primary endpoints. So when you say they don't "have" to be the same, does it make sense to compute the sample size with an $\alpha=0.025$ and then perform the t-test with an $\alpha=0.05$ given that this strategy was planned from the begining? This means we were conservative while computing the sample size. Wasn't this enough? Do we still have to keep a $\alpha=0.025$ for the t-test? $\endgroup$ – user3631369 May 30 '16 at 17:22
  • $\begingroup$ I mean what would "technically" happen in terms of interpretation if the $\alpha$ changes? From a small $\alpha$ to a bigger one and the opposite? $\endgroup$ – user3631369 May 30 '16 at 17:28
  • $\begingroup$ You already have your sample, so with whatever $\alpha$ you choose for your test, $n$ isn't going to change in your calculation of a t-statistic. Accepting a larger $\alpha$ means that you're willing to accept a greater likelihood of a false positive, and that your standard errors will be larger, but there are far better discussions of $\alpha$ and p-values on this site than can be summarized in a comment. Point being that you can use a different $\alpha$ in your test if you have a good reason, but in your write up, you should flag any deviations from your initial hypothesis/design. $\endgroup$ – 5ayat May 30 '16 at 20:47
0
$\begingroup$

The $\alpha$ is usually 0.05 unless multiple tests correction is applied.

(The problem is usually in the effect size (in other words, in $\epsilon$). If the effect size is not exactly known, the sample size calculations are only approximate. Even after the experiment is done you can't know the power.)

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.