# When is probability of type-I error less than the level of significance?

My book mentions

$$\textrm{Probability of Type I Error}\le \textrm{Level of Significance}= \alpha$$

Now, I bore in mind that $\alpha$, the level of significance is described by $$\mathrm P(t\in \omega|H_0)= \alpha$$ where $\omega = \textrm{critical region}\,.$

I could get the sense of probability of Type I error equals $\alpha$ but when it is less than $\alpha\;?$

I'm not getting that.

Is it a textbook-erratum? If not, when does this case arise?

• It should be $\omega=$critical region, no? – Christoph Hanck Mar 23 '16 at 12:26
• It could happen with discrete distributions, e.g. with hypothesis tests based on Binomial random variables – user83346 Mar 23 '16 at 12:32
• @fcop: Would I be wrong if I say answers are not allowed in comment ;P Please make it an answer; I would be grateful. – user74724 Mar 23 '16 at 12:37
• @ChristophHanck: Sorry; fixing it. – user74724 Mar 23 '16 at 12:37
• To complement @fcop's comment and answer: you have to distinguish the nominal (desired, or "target") significance level and the effective significance level. – Stéphane Laurent Mar 23 '16 at 14:16

It is not a typo and for example arises when testing composite hypotheses, such as a one-sided test of the mean of a distribution, as in $H_0:\mu\leq0$ against $H_0:\mu>0$. We choose the critical value (equivalently, $\omega$) so that the probability of a type-I error is still just $\alpha$ even when $\mu$ is very close to being in the set specified by $H_1$, i.e., when $\mu=0$, i.e., when the probability of a type-I error is highest.

Now if, say, the true $\mu$ equals $-5$, the probability of obtaining a sample such that $t\in\omega$ will be much smaller. For example, suppose we sample from a $N(\mu,1)$ distribution. Then, our t-statistic for known $\sigma$ and testing against positive $\mu$ would simply be $t=\sqrt{n}\bar{x}$ and follow a $N(\mu,1)$ distribution, too.

The plot below shows the probability of rejecting $H_0$, i.e., the power function. The region to the left of the vertical dashes is where the null is true, and you can see that the rejection probability is .05 only at $\mu=0$.

mu <- seq(-3,3,by=.02)
plot(mu,1-pnorm(1.645,mean=mu),type="l",lwd=2,col="sienna3")
abline(h=.05,lty=2)
abline(v=.0,lty=2)

• Thanks, sir, for the answer. So, level of significance is a much broader term than probability of Type I error, I guess? – user74724 Mar 23 '16 at 12:00
• It is the supremum over the rejection probabilities taken over all parameters in the set specified by $H_0$. – Christoph Hanck Mar 23 '16 at 12:11

The probability of a type I error may be stricly smaller than the significance level $\alpha$ for discrete distributions. E.g. if you want to test whether a coin is upward biased i.e. $H_1: p > 0.5$ and we test this against $H_0: p \le 0.5$.

Then the critical region is in the right queue, assume that we observe 10 tosses. Choosing $\alpha=0.05$ the critical region is $\omega = \{9,10\}$ and $P(t\in\omega)=0.0107$

• @fcop (+1) you surely mean $H_0: \le 0.5$ – peuhp Mar 23 '16 at 20:11
• @peuhp: you are right, I edited the answer – user83346 Jun 20 '16 at 16:28