Podcast #128: We chat with Kent C Dodds about why he loves React and discuss what life was like in the dark days before Git. Listen now.
2 edited title
| link

The Errorerror of first and second order and the asymmetry of statistical tests

source | link

The Error of first and second order and the asymmetry of statistical tests

I am aware that the question about the asymmetry of hypothesis testing is not a new question (and I have done my homework and read about it quite a bit). However, I still do have two questions I would like to adress.

First just my understanding of the asymmetry in hypothesis testing: I understand this problem as a problem generally inherent in scientific prognosis, see Karl Popper's distinction between verification and falsification: At best we can falsify but there is no ultimate verification.

My questions:

  • How are the significance level and the power of a statistical test related to this asymmetry?

Given a parameter space $\Theta $ and the null-hypothesis $\Theta_{0} \in \Theta$ and the alternative $\Theta_{A} \in \Theta$. Assuming the parameter p is in $\Theta_{0}$, the probability of nullhypothesis to be rejected is given by the significance level $\alpha$. But, if we go for a certain siginifcance level $\alpha$, still we have not necessarily decided yet about the power $\beta(p)$ of our test, i.e. the probabiliy that the nullhypotheis is accepted given p $\in \Theta_{A}$ (although I am aware that often it is the case $\beta = 1 - \alpha$).

So, where in the choice of $\alpha$ and $\beta$ is the asymmetry of hypothesis testing reflected?

  • Given T is our test statistic, can the asymmetry in hypothesis testing also be explained as the fact that we compute $P( T \geq t | H_{0})$, the probability to observe the data if that given the null-hypothesis holds and not $P( H_{0} | T \geq t)$, the probability that the null-hypothesis is true given the observations?

Thanks Pegah