• Reducing Type 1 error will always result in increasing the Type 2 error

This statement is false. I understand the definitions of Type 1 and Type 2 errors. What I understand is that there is, in fact, a trade-off between the errors. However, for this particular statement, a counterexample I across was given was a test procedure which never rejects $H_o$. What I could not figure out is how a certain probability of a particular type of error could mess with the 'trade-off'.

  • $\begingroup$ A procedure that never rejects the null cannot have its Type I error reduced. Thus, the quoted statement is logically true in that case. Having said that, I can think of many counterexamples, but they are of an entirely different nature. Moreover, they do not prove the quotation false, because the quotation is intended to convey a general idea and adopts many implicit assumptions. In light of this, what are you trying to ask? $\endgroup$ – whuber Nov 27 '18 at 17:59
  • $\begingroup$ @whuber I would like to understand the statement in context of a counterexample for the same if possible. The example I provided, like you said, proves the statement true. I understand now. Could you provide a counterexample which contradicts the statement (not vacuously)? That said, is the statement even False? $\endgroup$ – S.Rana Nov 27 '18 at 18:28
  • $\begingroup$ It all depends on what one does to reduce the Type 1 error. Presumably the operation consists solely of changing the critical region of a simple hypothesis test involving a continuous test statistic. Thus, by relaxing the criteria--considering other ways to change error rates, looking at composite hypotheses, and/or considering problems with discrete distributions, you are likely to find "counterexamples." $\endgroup$ – whuber Nov 27 '18 at 19:32

For a typical hypothesis test with simple null hypothesis $H_0$ and simple alternative hypothesis $H_1$, and a continuous random variable $X$ as the observation, let the pdf of $X$ be denoted by $f_0(x)$ whenever the null hypothesis is true and by $f_1(x)$ whenever the alternative hypothesis is true. If $\Gamma_1$ denotes the region such that the null hypothesis is rejected whenever $X \in \Gamma_1$, then the Type I error probability -- the probability of rejecting the null when in fact the null hypothesis is true -- is $$P(E_I) = P\{X \in \Gamma_1 \mid H_0 ~\text{is true}\} = \int_{\Gamma_1} f_0(x) \, \mathrm dx\tag{1}$$ while the Type II error probability -- the probability of failing to reject the null when in fact the null is false -- is $$P(E_{II}) = P\{X \in \Gamma_1^c \mid H_0 ~\text{is false}\} = 1 - \int_{\Gamma_1} f_1(x) \, \mathrm dx.\tag{2}$$

Now, one way of reducing $P(E_I)$ is to shrink the size of $\Gamma_1$ to $\Gamma_1^\prime \subset \Gamma_1$ where we assume that $f_0(x)$ is nonzero in the region $\Gamma_1 - \Gamma_1^\prime$ so that the integral in $(1)$ is over a smaller region and we can be sure that $P(E_I)^\prime < P(E_I)$. But then, the integral in $(2)$ is also over a smaller region and so $P(E_{II})^\prime \geq P(E_{II})$. It is tempting to insist that it must be the case that $P(E_{II})^\prime$ is strictly larger than $P(E_{II})$ but this does not hold if $f_1(x)$ is identically $0$ on $\Gamma_1 - \Gamma_1^\prime$ and so shrinking $\Gamma_1$ has no effect on the Type II error probability. Note that on $\Gamma_1 - \Gamma_1^\prime$, $f_0(x)$ is nonzero while $f_1(x) = 0$, that is, the likelihood ratio $\displaystyle\frac{f_1(x)}{f_0(x)}$ equals $0$ and so why the original test perversely chose to include these points in $\Gamma_1$ is beyond me: perhaps only to show that there are exceptions to the rule that

Reducing Type I error will always result in increasing the Type II error


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.