1
$\begingroup$

I know what the significance level is and how hypothesis testing works, so this is primarily a terminological question.

In hypothesis testing, the so-called "significance level" $\alpha$ equals the probability of falsely rejecting a true null hypothesis. If this value is particularly low for a given test for which the null hypothesis has been rejected, this means that the result is very unlikely to have been obtained by chance alone. This yields the very counterintuitive result that when considering a given hypothesis test and the associated decision of rejection or non-rejection of the null hypothesis, a lower significance level means a more highly significant result. The lower the significance level, the higher the significance of the result...

Intuitively, one would expect "significance level" to mean something similar to "confidence level." Unfortunately, it means the opposite. I find that this confuses undergraduates quite a bit and it would be helpful to have a good explanation for them as to how naming $\alpha$ the significance level makes sense.

So, what are the historical, statistical, or mathematical reasons that the significance level inversely related to the confidence level?

$\endgroup$
4
$\begingroup$

It may help to recall that there is no such thing as a "more significant" or "more highly significant" effect. Significance is binary, not scalar: either an effect is declared significant, or it isn't. The whole idea of significance testing is based on this kind of binary decision-making. So $α$ is called the significance level because it's the maximum $p$ allowed for significance. Confidence intervals and their associated confidence levels have completely different motivations, formulations, and interpretations, and not every good confidence interval corresponds to a good significance test, nor vice versa, so it shouldn't be too surprising that the conventions of the two approaches aren't wholly compatible.

This said:

  • In my opinion, "significance level" is a somewhat sloppy term; "type-I error rate" is a better choice for technical material.
  • Technical treatments of confidence intervals tend to write the confidence level as $1 - α$ to underscore the analogy to significance testing: the smaller the $α$, the more conservative the analysis.
$\endgroup$
  • $\begingroup$ +1 At least there's no "more significant" from a Neyman-Pearson perspective. Fisher might have been more equivocal about the notion. $\endgroup$ – Glen_b -Reinstate Monica Jul 18 '17 at 15:20
  • 1
    $\begingroup$ @Glen_b Yes, but I tend to think that the word "significant" belongs to the Neyman–Pearson tradition. In Fisherian testing, you don't need to call anything significant; you just have more or less evidence against the null hypothesis. $\endgroup$ – Kodiologist Jul 18 '17 at 15:34
  • 1
    $\begingroup$ Ah, yes, at least with respect to the phrase "more significant" you may be right. He certainly used terms like "tests of significance" though. Actually here's a quote: "Critical tests of this kind may be called tests of significance, and when such tests are available we may discover whether a second sample is or is not significantly different from the first." ... that's from 1925. $\endgroup$ – Glen_b -Reinstate Monica Jul 18 '17 at 15:37
  • $\begingroup$ @Glen_b Shoot, that phrasing is confusing at best; without a discrete cutoff, it doesn't make much sense to say "whether" something "is or is not significantly different". $\endgroup$ – Kodiologist Jul 18 '17 at 15:45
  • $\begingroup$ While I couldn't find anything of him saying "more significant" he does come a little closer to it on occasion, such as No isolated experiment, however significant in itself, can suffice for the experimental demonstration of any natural phenomenon ... --- the phrasing there at least suggests he's thinking of differing degrees of significance. $\endgroup$ – Glen_b -Reinstate Monica Jul 18 '17 at 15:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.