1
$\begingroup$

All the statistic's courses state that there is a tradeoff between the significance level ($\alpha$) and the probability of the Type II error ($\beta$). Specifically, for a fixed sample size, one cannot find a critical region that yields arbitrarily small values of $\alpha$ and $\beta$, and thus the consensus establishes that the statistician should fix the value of $\alpha$ and find the most powerful (greater value of $1 - \beta$) critical region according to that significance level.

I have not seen a proof of the $\alpha-\beta$ tradeoff in a general framework. Could you please provide me with the insights of a proof?

Thanks in advance

$\endgroup$
1
$\begingroup$

When you increase $\alpha$, you increase the probability of rejecting true nulls, and the way to achieve that is by making the rejection region larger. When that happens, the cases in which we reject can only increase (or in any case not decrease), be it when the null is true or when it is false. Then, $\beta$, the probability of a false acceptance, must decrease (or in any case not increase).

Of course, the reasoning works mutatis mutandis with decreasing $\alpha$.

To write that up a little more formally, denote by $\cal{T}$ the support of the test statistic $t$. A significance level $\alpha$ is associated with a critical value $c_\alpha$. W.l.o.g.$^*$ (take the absolute value of $t$ if the original test statistic implied a rejection when it took value less than $-c_\alpha$ or bigger than $c_\alpha$), we reject if $t>c_\alpha$.

This defines the rejection region $R_\alpha:=\{t\in\cal{T}:t>c_\alpha\}$.

We then obtain the rejection probability, given a true parameter $\theta$, as $$ P_\theta(t\in R_\alpha) $$ If we test the null $\theta=\theta_0$ and it is actually true (setting aside issues like errors in rejection probabilities stemming from using asymptotic tests and/or nuisance parameters), $$ P_{\theta_0}(t\in R_\alpha)=\alpha $$ When $\theta\neq\theta_0$, $$ P_{\theta}(t\in R_\alpha)=1-\beta, $$ the power of the test.

Now, if we want to make $\alpha$ smaller, we must make make $R_\alpha$ smaller, $$\alpha_1<\alpha_2\Rightarrow R_{\alpha_1}\subset R_{\alpha_2}.$$ Then, for any $\theta$, the probability that a random variable takes realizations in a smaller set is smaller (I am handwaiving over strict and weak inequalities here) than that it takes realizations in the larger set, $$ P_{\theta}(t\in R_{\alpha_1}) < P_{\theta}(t\in R_{\alpha_2}) $$

$^*$: Actually, here we have a little scope for coming up with pathological cases to construct other cases: If we only require rejections with probability $\alpha$ under the null, we could also allocate the first, larger, rejection region of, say, a right-tailed test $\theta\leq\theta_0$ to the left tail of the null distribution. The probability that $t$ takes realizations in that region will be even smaller than $\alpha$ if the null is false in the sense that $\theta>\theta_0$. If we now decrease $\alpha$ and push that smaller rejection region to where it belongs, the right tail of the null distribution, power will be larger than before.

Of course, no sane person would have picked the first rejection region in that way, because a sensible auxiliary requirement for a rejection region is that it contains values which we would not expect to occur frequently if the null is actually true.

Now, when the null $\theta\leq\theta_0$ is true, small values of $t$ (i.e., values in the left tail of the least favorable distribution $\theta=\theta_0$) may not be evidence against the null at all.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for your answer, although I was looking for something more general. Your proof relies on the fact that you are taking critical regions with the form $\{t\in\mathcal{T}\}$ and, as you said, one can come up with different critical regions, even real bizarre ones. $\endgroup$ – Abel Mar 15 '19 at 13:02
  • $\begingroup$ That is true, and I believe that it covers how hypothesis testing is (and should be) done. As I try to spell out in my footnote, if you are willing to consider such "bizarre" regions, you can't prove the statement, because it is no longer true. $\endgroup$ – Christoph Hanck Mar 15 '19 at 13:11
0
$\begingroup$

let

$H_0:\mu=\mu_0$

$H_1:\mu=\mu_1$ ($\mu_0 \neq \mu_1$)

$\phi(x)=\left\{ \begin{array}{cc} 1 & x\in C \\ 0 & x\in C^{\prime} \end{array} \right.$

$C$ is rejection area. note that $\alpha + \beta \leq 1$ (Corollary 3.2.1 in Testing Statistical Hypotheses (E.L. Lehmann Joseph P. Romano). for different values of $\alpha$ and $\beta$, because with increasing $\alpha$, $\beta$ should decreasing, to holding $\alpha + \beta \leq 1$.

if we plot $(\alpha, \beta)$ it will help.

define

$N=\{(\alpha , \beta) | for \hspace{.5cm} any \hspace{.5cm} \phi^{\prime}(x) \hspace{.5cm} s.t \hspace{.5cm} E_{H_0}(\phi^{\prime}(x)) =\alpha \hspace{.5cm} E_{H_1}(\phi^{\prime}(x))=1-\beta \}$.

$N$ is convex and includes $(0,1)$ and $(1,0)$ and around $(.5,.5)$ is Symmetric(See Testing Statistical Hypotheses (E.L. Lehmann Joseph P. Romano, with one difference that is take $N=(\alpha,1-\beta)$ and by 90 degree transformation you get $(\alpha,\beta)$).

N will be something like N

so we can not optimize $\alpha $ and $\beta$ with together and according the shape of $N$ for any level of $\alpha$ we take $\phi$ that have minimum $\beta$.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.