According the p-value Wikipedia page:

When the p-value is calculated correctly, this test guarantees that the Type I error rate is at most $\alpha$.

However further down the page this formula is given:

$$\Pr(\mathrm{Reject}\; H|H) = \Pr(p \leq \alpha|H) = \alpha$$

Assuming 'type 1 error rate' = $\Pr(\mathrm{Reject}\; H|H)$ this suggest that the type 1 error rate is $\alpha$ and not 'at most $\alpha$'. Otherwise the formula would read:

$$\Pr(\mathrm{Reject}\; H|H) \leq \alpha$$

Where is my misunderstanding?


When the "null hypothesis" includes more than one state of nature, the actual false positive rate (FPR) may vary with that state. All we can do is guarantee a limit on the FPR no matter what that state of nature might be--but we cannot always guarantee the FPR actually equals $\alpha$.

(There are other reasons why the FPR might not actually equal its targeted value $\alpha$, such as when the test statistic is discrete. These situations usually can be cured by using randomized decision procedures. As such they do not provide any fundamental insight into the question.)

Consider the classical one-tailed test where the statistic $X$ is assumed to have a Normal distribution of unknown mean $\mu$ and (for simplicity) known standard deviation $\sigma$. $\mu$ is to be compared to $0$. The null hypothesis is $H_0:\mu \ge 0$ while the alternative hypothesis is $H_A:\mu \lt 0$. The rejection region therefore is of the form

$$\mathcal{R}(\alpha) = (-\infty, Z_\alpha]$$

where $Z_\alpha$ is chosen so that the chance of observing a statistic in this region is at most $\alpha$:

$$\alpha =\sup\left(\Pr(X \in \mathcal{R}(\alpha))\right)\tag{1}.$$

Under the assumptions, this probability is given by the Normal distribution function $\Phi$:

$$\Pr(X \in \mathcal{R}(\alpha)) = \Phi\left(\frac{Z_\alpha-\mu}{\sigma}\right)\tag{2}.$$

This probability depends on the unknown value of $\mu$. Therefore we cannot guarantee that it actually equals $\alpha$. Indeed, for large $\mu$, $(2)$ is practically zero. We have to cover all our bases, though, and guarantee that as long as $\mu$ is consistent with the null hypothesis, the false positive rate $(1)$ will not exceed $\alpha$.

  • 1
    $\begingroup$ @JackPierce-Brown The formula is correct for point null hypothesis and for continuous test statistic. That's what must be assumed in the Wikipedia article, but is probably not spelled out. (+1) $\endgroup$ – amoeba Feb 2 '17 at 21:47
  • 1
    $\begingroup$ @Amoeba is right. Note, in addition, that only a few practical tests truly involve point Null hypotheses. Even the classical Student t-test of $H_0:\mu=0$ vs $H_A:\mu \gt 0$ is not a point Null, because there are manifold possibilities for the unknown value of the parameter $\sigma$ even though the null pins down the value of $\mu$. $\endgroup$ – whuber Feb 2 '17 at 21:51
  • 1
    $\begingroup$ @whuber Hmm, your t-test example is puzzling. Can you elaborate? I thought $H_0=0$ is a point null, because $0$ is a point, and $\sigma$ does not enter the null hypothesis. If it is not a point null, does it mean that type I error rate is not equal to $\alpha$? I would have thought it should be equal to $\alpha$ no matter what $\sigma$ is. $\endgroup$ – amoeba Feb 2 '17 at 21:58
  • 2
    $\begingroup$ @Amoeba $\sigma$ very much is part of the null hypothesis. Rigorously, the parameter space is $$\Theta = \{(\mu,\sigma)\mid \mu\in\mathbb{R},\,\sigma \ge 0\}.$$ The null hypothesis is the subset $$H_0=\{(\mu,\sigma)\mid \mu=0,\sigma\ge 0\} \subset\Theta.$$ It's not a single state of nature. But perhaps this isn't the best possible example, because the distribution of the $t$ statistic does not depend on $\sigma$: that is why a constant FPR is possible. $\endgroup$ – whuber Feb 2 '17 at 22:02
  • 1
    $\begingroup$ Interesting. I see. $\endgroup$ – amoeba Feb 2 '17 at 22:03

It's a sneaky issue. If you have continuous data, and you treat them appropriately, then $\Pr(p \leq \alpha|H_0) = \alpha$. However, when your data are discrete, it may not be possible for $p = \alpha$. Consider binomial data on whether a coin is fair, with 5 coin flips, the possible one-sided p-values are:

> pbinom(0:5, size=5, prob=.5)
[1] 0.03125 0.18750 0.50000 0.81250 0.96875 1.00000

Only $0$ heads could yield a type I error, and the probability associated with that is $\approx 0.03$. So then the type I error rate would be held to "at most $α$", but not equal to $\alpha$.

On the other hand, there are (invalid) analysis strategies that lead to type I error rates that are greater than $\alpha$, even when $p<\alpha$ (e.g., stepwise selection routines).

I have a fuller discussion here: Comparing and contrasting, p-values, significance levels and type I error


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.