11
$\begingroup$

In an elementary statistics class that I was a TA for, the professor stated that as the probability of a type I error $\alpha$ increases, the probability of a type II error $\beta$ decreases, and the converse is true as well. So this suggests to me that $\rho_{\alpha, \beta} < 0$.

But how would one prove this for a general hypothesis test? Is the statement even true in general?

I could try a specific case (say $H_0: \mu = \mu_0$ and $H_1: \mu < \mu_0$) but obviously, that's not general enough to handle this question.

$\endgroup$
13
$\begingroup$

These quantities ($\alpha$ and $\beta$) are not random variables, so I hesitate to speak of their Pearson correlation; I'm not sure in what sense that would apply.

The two are negatively related in the sense that, reasonably generally speaking (but see below*) - and holding other things (like sample size and the effect size at which you compute $\beta$) equal - if you change $\alpha$, then $\beta$ will move the opposite direction (specifically, in typical situations, $\beta$ is a function of $\alpha$; specify enough quantities to determine $\beta$ and it will depend on $\alpha$ -- and that relationship will, in most reasonable situations - the kind you'd want to use in an actual test - be negatively dependent).

Consider, for example, some power curve. Moving $\alpha$ will push the power curve ($1-\beta$) up or down with it, so $\beta$ at some point on the curve (which is the distance between the curve and 1) decreases as $\alpha$ increases. Here's an example with a two-tailed test (say a t-test).

enter image description here

The one-tailed case is similar, but you'd focus on the right-half of the above picture (the two curves in the left half of the picture would tail down toward zero)


* there are some situations where this doesn't have to be the case. Consider testing for a uniform(0,1) via a Kolmogorov-Smirnov test.

Let's consider the possibility that instead we have a uniform on $(0,1+\epsilon)$ $^\dagger$ (or indeed, any distribution with some probability outside the unit interval).

If I observe a value that doesn't lie in (0,1), the Kolmogorov-Smirnov test doesn't necessarily reject the null. But I can make a second test, (let's call it the KS* test), which is like the Kolmogorov-Smirnov, except that when we observe a value outside (0,1) we also reject the null whether or not the usual statistic reaches the critical value.

Then for any alternative which has any probability outside (0,1) we have reduced the Type II error rate (from that for the ordinary KS test) without changing $\alpha$ at all.

$\dagger$ (it's not typically a great idea to use a K-S in that case, so if you know it's a possibility, you need to think carefully about alternatives)

$\endgroup$
3
$\begingroup$

Let $X$ denote the observation with density $f_0(x)$ or $f_1(x)$ according as Hypothesis $H_0$ or $H_1$ is true. Let $\Gamma_0$ and $\Gamma_1$ denote the decision regions. Thus, $\Gamma_0 \cap \Gamma_1 = \emptyset$, $\Gamma_0 \cup \Gamma_1 = \mathbb R$ and the decision is that $H_i$ is true iff $X \in \Gamma_i$. Then, the Type I and Type II error probabilities are $$\begin{align} P(\text{Type I error}) &= \int_{\Gamma_1}f_0(x)\,\mathrm dx\tag{1}\\ P(\text{Type II error}) &= \int_{\Gamma_0}f_1(x)\,\mathrm dx.\tag{2} \end{align}$$ Consider two other decision regions $\Gamma_0^\prime$ and $\Gamma_1^\prime$ such that $\Gamma_1 \subset \Gamma_1^\prime$ and $\Gamma_0^\prime \subset \Gamma_0$. Now, $$\int_{\Gamma_1^\prime}f_0(x)\,\mathrm dx \geq \int_{\Gamma_1}f_0(x)\,\mathrm dx$$ since the integral is over a larger set, which means that the new decision rule has a larger Type I error probability. But note also that $$\int_{\Gamma_0^\prime}f_1(x)\,\mathrm dx \leq \int_{\Gamma_0}f_1(x)\,\mathrm dx$$ because the integral is over a smaller set, and so the new decision rule has a smaller Type II error probability.

$\endgroup$
1
$\begingroup$

The relationship you are looking at between $\alpha$ and $\beta$ is true with respect to the activity they intended you to be thinking of at the time: adjusting the critical-value you use for accepting or rejecting a hypothesis. If you make it harder to achieve a false-positive, you naturally have to make it easier to achieve a false-negative. This website shows the relationship between $\alpha$ and $\beta$ graphically.

The relationship is not true for all activities. As an obvious example, if you increase the number of samples in your test, you can decrease $\alpha$ and $\beta$ simultaneously. The relationship is only guaranteed for when you are adjusting critical values

$\endgroup$
  • 1
    $\begingroup$ "The relationship is only" - it seems the tail end of your answer was cut off? $\endgroup$ – Silverfish Jan 9 '15 at 8:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.