Type 1 and Type 2 errors trade-off

• Reducing Type 1 error will always result in increasing the Type 2 error

This statement is false. I understand the definitions of Type 1 and Type 2 errors. What I understand is that there is, in fact, a trade-off between the errors. However, for this particular statement, a counterexample I across was given was a test procedure which never rejects $$H_o$$. What I could not figure out is how a certain probability of a particular type of error could mess with the 'trade-off'.

• A procedure that never rejects the null cannot have its Type I error reduced. Thus, the quoted statement is logically true in that case. Having said that, I can think of many counterexamples, but they are of an entirely different nature. Moreover, they do not prove the quotation false, because the quotation is intended to convey a general idea and adopts many implicit assumptions. In light of this, what are you trying to ask? – whuber Nov 27 '18 at 17:59
• @whuber I would like to understand the statement in context of a counterexample for the same if possible. The example I provided, like you said, proves the statement true. I understand now. Could you provide a counterexample which contradicts the statement (not vacuously)? That said, is the statement even False? – S.Rana Nov 27 '18 at 18:28
• It all depends on what one does to reduce the Type 1 error. Presumably the operation consists solely of changing the critical region of a simple hypothesis test involving a continuous test statistic. Thus, by relaxing the criteria--considering other ways to change error rates, looking at composite hypotheses, and/or considering problems with discrete distributions, you are likely to find "counterexamples." – whuber Nov 27 '18 at 19:32

For a typical hypothesis test with simple null hypothesis $$H_0$$ and simple alternative hypothesis $$H_1$$, and a continuous random variable $$X$$ as the observation, let the pdf of $$X$$ be denoted by $$f_0(x)$$ whenever the null hypothesis is true and by $$f_1(x)$$ whenever the alternative hypothesis is true. If $$\Gamma_1$$ denotes the region such that the null hypothesis is rejected whenever $$X \in \Gamma_1$$, then the Type I error probability -- the probability of rejecting the null when in fact the null hypothesis is true -- is $$P(E_I) = P\{X \in \Gamma_1 \mid H_0 ~\text{is true}\} = \int_{\Gamma_1} f_0(x) \, \mathrm dx\tag{1}$$ while the Type II error probability -- the probability of failing to reject the null when in fact the null is false -- is $$P(E_{II}) = P\{X \in \Gamma_1^c \mid H_0 ~\text{is false}\} = 1 - \int_{\Gamma_1} f_1(x) \, \mathrm dx.\tag{2}$$
Now, one way of reducing $$P(E_I)$$ is to shrink the size of $$\Gamma_1$$ to $$\Gamma_1^\prime \subset \Gamma_1$$ where we assume that $$f_0(x)$$ is nonzero in the region $$\Gamma_1 - \Gamma_1^\prime$$ so that the integral in $$(1)$$ is over a smaller region and we can be sure that $$P(E_I)^\prime < P(E_I)$$. But then, the integral in $$(2)$$ is also over a smaller region and so $$P(E_{II})^\prime \geq P(E_{II})$$. It is tempting to insist that it must be the case that $$P(E_{II})^\prime$$ is strictly larger than $$P(E_{II})$$ but this does not hold if $$f_1(x)$$ is identically $$0$$ on $$\Gamma_1 - \Gamma_1^\prime$$ and so shrinking $$\Gamma_1$$ has no effect on the Type II error probability. Note that on $$\Gamma_1 - \Gamma_1^\prime$$, $$f_0(x)$$ is nonzero while $$f_1(x) = 0$$, that is, the likelihood ratio $$\displaystyle\frac{f_1(x)}{f_0(x)}$$ equals $$0$$ and so why the original test perversely chose to include these points in $$\Gamma_1$$ is beyond me: perhaps only to show that there are exceptions to the rule that