The error of first and second order and the asymmetry of statistical tests

Question

I am aware that the question about the asymmetry of hypothesis testing is not a new question (and I have done my homework and read about it quite a bit). However, I still do have two questions I would like to adress.

First just my understanding of the asymmetry in hypothesis testing: I understand this problem as a problem generally inherent in scientific prognosis, see Karl Popper's distinction between verification and falsification: At best we can falsify but there is no ultimate verification.

My questions:

• How are the significance level and the power of a statistical test related to this asymmetry?

Given a parameter space $\Theta $ and the null-hypothesis $\Theta_{0} \in \Theta$ and the alternative $\Theta_{A} \in \Theta$. Assuming the parameter p is in $\Theta_{0}$, the probability of nullhypothesis to be rejected is given by the significance level $\alpha$. But, if we go for a certain siginifcance level $\alpha$, still we have not necessarily decided yet about the power $\beta(p)$ of our test, i.e. the probabiliy that the nullhypotheis is accepted given p $\in \Theta_{A}$ (although I am aware that often it is the case $\beta = 1 - \alpha$).

So, where in the choice of $\alpha$ and $\beta$ is the asymmetry of hypothesis testing reflected?

• Given T is our test statistic, can the asymmetry in hypothesis testing also be explained as the fact that we compute $P( T \geq t | H_{0})$, the probability to observe the data if that given the null-hypothesis holds and not $P( H_{0} | T \geq t)$, the probability that the null-hypothesis is true given the observations?

Thanks Pegah

Answer 1

A couple of corrections first. The errors that you have in mind are called "type I" and "type II" errors, after Neyman and Pearson discussed them as "errors of the first kind" and "errors of the second kind". It is often helpful to think of them as 'false positive' and 'false negative' errors. (I suspect that first order and second order mean something else.) For completeness I will say that a type I error occurs when the null hypothesis is discarded despite it being true. A type II error is where the null is accepted despite being false.

You are incorrect in your assertion that the type II error rate is often one minus the type I error rate. That is an unlikely proposition and it does not follow from any aspect the nature of hypothesis tests, and in practice the power is often much lower than alpha (the 'size' of the test). Generally the power is described as one minus the type II error rate, beta.

Now, the asymmetry. The maximal type I error rate is set in advance of the experiment by simply choosing how extreme the test statistic has to be in order to discard the null hypothesis. It is controlled by the experimenter. Note that I said 'maximal' in that sentence. The actual type I error rate depends on the alpha chosen and the proportion of experiments in which the null is actually true.

In contrast, type II error rates are determined by several things: the magnitude of discrepancy between the null hypothesised population and the true population (that is not the alternative hypothesis!); the sample size; and the test efficiency. And the proportion of the experiments conducted with a false null hypothesis (not the same as a true alternative).

The alternative is not true just because the null is false because the alternative is usually a specific hypothesis in Neyman-Pearsonian hypothesis testing, rather than being just 'not the null'.

The asymmetry can be seen in the longer list of determinants of the rate of type II errors.