7
$\begingroup$

As a convention, we have a lot of studies whose significance level is $0.05$ and a power of $0.8$. However, it is extremely rare to find a study whose $\alpha = 0.2$ with a power of $0.95$.

From my understanding, after an experiment has been conducted, the significance level doesn't matter at all if the result is non-significant, because in this case, we are considering whether it makes sense to accept the null, and all we care about is the power. Similarly, if the result is significant, then the significance level becomes your evidence, while the power of the test makes absolutely zero difference. (By "doesn't matter", I mean "doesn't for the purpose of this experiment". Both significance level and power should be important for meta-studies, so please report both in your publication!)

If I'm correct, then the null and the alternative are to some extent symmetrical: the null hypothesis doesn't inherently require more protection. If you want to prove the alternative, say "this new drug has an effect on the patients", then use a very small $\alpha$ and moderately high power. On the other hand, when you want to prove the null, for example in a normality test, then you should choose a moderately small $\alpha$ and very high power, so that you can confidentially accept the null.

Why are experiments with moderately small $\alpha$ and very high power so rare?

$\endgroup$
  • 3
    $\begingroup$ Because the cultural convention of $\alpha=0.05$ is strongly established? $\endgroup$ – Ben Bolker Jul 24 at 22:53
  • 2
    $\begingroup$ We see 5% all over, but not for any good reason; in many situations we should certainly consider smaller (and in some cases, larger) significance levels -- and in perhaps many more situations still, reconsider whether a significance test is actually the right tool for the job (it is often not, but if all you have in your toolbox is a hammer...). Fisher generally regarded 5% as essentially the weakest evidence he'd even consider paying attention to (and he was a stickler for replication of experiments on top of that), but for some reason that ended up becoming seen as a standard. $\endgroup$ – Glen_b -Reinstate Monica Jul 25 at 0:36
  • 2
    $\begingroup$ All I can say is if you are able to read some of what Fisher wrote about it, you'll see how important he saw replication as part of the process. I don't have a reference to hand but most of his papers (and comments on other papers) are publicly available. (His books may well be better sources, if not as readily viewed). As an example of its importance, see this paper ... which says (first sentence of 4th paragraph "Three fundamental experimental design principles attributed to Fisher are randomization, replication, and blocking". $\endgroup$ – Glen_b -Reinstate Monica Jul 25 at 1:43
  • 2
    $\begingroup$ That paper references Fisher's "Design of Experiments, 6ed". However, that mostly discusses the use of replication by the experimenter (which serves several important purposes); he also saw replication by others as important when trying to arrive at facts in the face of uncertainty as well. $\endgroup$ – Glen_b -Reinstate Monica Jul 25 at 2:08
  • 1
    $\begingroup$ Because of multiple testing and all the various other biases not accounted for, the false discovery rate tends to be much larger than $\alpha$. At the same time, the cost (and risk) of a false positive tend do be much worse. So at 0.2, in reality over 50% may turn out to be false in a more detailed but costly experiment. False negatives often don't cost much, in particular if you assume someone will independently test the same hypothesis again at a later stage. $\endgroup$ – Anony-Mousse Jul 25 at 8:41
5
$\begingroup$

Why are experiments with moderately small $\alpha$ and very high power so rare?

This is all a bit relative, but one could certainly argue that the significance level $\alpha = 0.05$ is already weak, and already constitutes a sacrifice made for higher power (e.g., relative to the significance level $\alpha = 0.01$ or other lower significance levels). While opinions on this will differ, my own view is that this is already a very weak significance level, so choosing it at all is already a trade-off to get higher power.

From my understanding, after an experiment has been conducted, the significance level doesn't matter at all if the result is non-significant, because in this case, we are considering whether it makes sense to accept the null, and all we care about is the power. Similarly, if the result is significant, then the significance level becomes your evidence, while the power of the test makes absolutely zero difference.

I can see why you might think this, but it is not really true. In classical hypothesis testing there is quite a complex and subtle interaction in these things. Remember that both the p-value and the power pertain to probabilities that condition on the true state of the hypotheses (the p-value conditions on the null, and the power conditions on the alternative). When you get your result from the data, you make an inference about the hypotheses, but you still don't know their true state. Thus, it is not really legitimate to say that you can completely ignore the "other half" of the test. Regardless of whether the result is statistically significant or not, the interpretation of that result is made holistically, with respect to all the properties of the test.

It is also worth noting that, for a fixed model and test, and a fixed sample size, the power function is a function of the chosen significance level. The chosen significance level determines the rejection region, which directly affects the power of the test. So again, there is a relationship between these things, and you cannot ignore "one half" of the properties of the test.

$\endgroup$
  • $\begingroup$ I would agree that $\alpha$ and $1 - \beta$ are negatively correlated, but by the time you conduct the experiment, its design must have been finalized, so at that point, $\alpha$ and $1 - \beta$ are already fixed parameters of the test. For a non-significant result, I can hardly understand how can you interpret it in terms of "the probability of the result being significant when the null is true". Are you considering the fact that a greater $\alpha$ implies a greater $p$? $\endgroup$ – nalzok Jul 25 at 0:44
  • $\begingroup$ A greater $\alpha$ does not imply a greater $p$. The latter is a function of the data, and is unaffected by $\alpha$. $\endgroup$ – Reinstate Monica Jul 25 at 0:50
  • $\begingroup$ I mean, conditioning on the result being non-significant, we have $p \ge \alpha$, so a greater $\alpha$ eliminates the possibility of a small $p$. This is the only way I can understand why $\alpha$ plays a role in interpreting a non-significant result. Is that what you are thinking about? $\endgroup$ – nalzok Jul 25 at 0:55
  • $\begingroup$ Roughly, but even then, the power function only looks at behaviour conditional on the alternative hypothesis being true. $\endgroup$ – Reinstate Monica Jul 25 at 23:02
4
$\begingroup$

This is more of an extended comment than an answer. One interesting perspective can be found in this blog post, a short citation:

... contends the word [significance] carried much less weight in the late 19th century, when it meant only that the result showed, or signified, something. Then, in the 20th century, significance began to gather the connotation it carries today, of not only signifying something but signifying something of importance. ...

If this is correct then Fisher can have meant with significant rather something like worthy of taking a note (mental or in lab notebook), worthy for further investigation or replication.

This psyarxiv paper proposing to reduce standard significance level (in psychology research) from 0.05 to 0.005 is further evidence that many see (rightly ...) that 0.05 is already a rather weak requirement.

$\endgroup$
1
$\begingroup$

Because type II errors are considered to be less of a problem than type I errors. Type I errors have greater implications for future research. Moreover, most of the time, experiments with high power are much more expensive.

But of course you can also question both the whole NHST framework and the way it is frequently misused by unaware researchers...

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.