16
$\begingroup$

What is the difference between the two and why must the level of significance be always higher than or equal to the size of the test?

$\endgroup$
2
  • 1
    $\begingroup$ I don't recognize the meaning of "size of a test." Perhaps you mean "size of a test statistic" such as F or T or Z. In that case the level of significance (p) is not necessarily higher or lower. Are you quoting from a particular source? If so, please include the quote and someone will no doubt help clarify it for you. $\endgroup$
    – rolando2
    Commented Mar 3, 2013 at 13:11
  • 5
    $\begingroup$ @rolando "Test size" is a standard term: see scholar.google.com/…. $\endgroup$
    – whuber
    Commented Jan 6, 2014 at 17:56

1 Answer 1

32
$\begingroup$

Suppose you have a random sample $X_1,\dots,X_n$ from a distribution that involves a parameter $\theta$ which assumes values in a parameter space $\Theta$. You partition the parameter space as $\Theta=\Theta_0\cup\Theta_1$, and you want to test the hypotheses $$ H_0 : \theta \in \Theta_0 \, , $$ $$ H_1 : \theta \in \Theta_1 \, , $$ which are called the null and alternative hypotheses, respectively.

Let $\mathscr{X}$ denote the sample space of all possible values of the random vector $X=(X_1,\dots,X_n)$. Your goal in building a test procedure is to partition this sample space $\mathscr{X}$ into two pieces: the critical region $\mathscr{C}$, containing the values of $X$ for which you will reject the null hypothesis $H_0$ (and, so, accept the alternative $H_1$), and the acceptance region $\mathscr{A}$, containing the values of $X$ for which you will not reject the null hypothesis $H_0$ (and, therefore, reject the alternative $H_1$).

Formally, a test procedure can be described as a measurable function $\varphi:\mathscr{X}\to\{0,1\}$, with the obvious interpretation in terms of the decisions made in favor of each of the hypotheses. The critical region is $\mathscr{C}=\varphi^{-1}(\{1\})$, and the acceptance region is $\mathscr{A}=\varphi^{-1}(\{0\})$.

For each test procedure $\varphi$, we define its power function $\pi_\varphi:\Theta\to[0,1]$ by $$ \pi_\varphi(\theta) = \Pr(\varphi(X)=1\mid\theta) = \Pr(X\in\mathscr{C}\mid\theta) \, . $$ In words, $\pi_\varphi(\theta)$ gives you the probability of rejecting $H_0$ when the parameter value is $\theta$.

The decision to reject $H_0$ when $\theta\in\Theta_0$ is wrong. So, for a given problem, you may want to consider only those test procedures $\varphi$ for which $\pi_\varphi(\theta)\leq\alpha$, for every $\theta\in\Theta_0$, in which $\alpha$ is some significance level ($0<\alpha<1$). Note that the significance level is a property of a class of test procedures. We can describe this class precisely as $$ \mathscr{T}_{\alpha} = \left\{ \varphi\in\{0,1\}^\mathscr{X} : \pi_\varphi(\theta)\leq\alpha, \textrm{for every}\; \theta\in\Theta_0\right\} \, . $$

For each individual test procedure $\varphi$, the maximum probability $\alpha_\varphi=\sup_{\theta\in\Theta_0}\pi_\varphi(\theta)$ of wrongly rejecting $H_0$ is called the size of the test procedure $\varphi$.

It follows directly from these definitions that, once we have established a significance level $\alpha$, and therefore determined the class $\mathscr{T}_{\alpha}$ of acceptable test procedures, each test procedure $\varphi$ within this class will have size $\alpha_\varphi\leq\alpha$, and conversely. Concisely, $\varphi\in\mathscr{T}_{\alpha}$ if and only if $\alpha_\varphi\leq\alpha$.

$\endgroup$
4
  • 1
    $\begingroup$ Wow. Thanks for all the effort you invested in this answer. $\endgroup$
    – asb
    Commented May 26, 2016 at 6:14
  • 1
    $\begingroup$ I came here to learn about size vs level and left understanding hypothesis testing better overall. Excellent combination of intuition and notation. $\endgroup$
    – jds
    Commented Oct 18, 2019 at 14:59
  • $\begingroup$ Thanks for the detailed answer. I see that this summary matches with what is found in some statistical theory books that I have. I notice that more basic books for business students consider just the case of simple null hypothesis and define the "level of significance" as the probability of type one error. So, I wonder whether this is due to some lack of rigour of such books or the definitions of "level of significance" and "size" are subject to changes from one textbook to another. $\endgroup$ Commented Jun 28, 2020 at 9:45
  • $\begingroup$ In other words, would it be wrong (or too different from the mainstream) to just define the level of significance or size as the maximum (or the sup of the) probability of type one error and do not make a distinction between "size" and "level of significance"? $\endgroup$ Commented Jun 28, 2020 at 9:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.