4
$\begingroup$

I understand that the p-value is defined as the probability to obtain a "more extreme" value of $w$ if $H_0$ is true, i.e.

$p=P(|W| > |w| \ |H_0 $is true$)$

and the "significance level" $\alpha$ is a threshold to decide if it is "admissible" that $w$ comes in null hypothesis setting, i.e.

$p < \alpha \Rightarrow$ reject $H_0$.

However, It is hard for me to understand the meaning of hypothesis test when $\alpha$ is interpreted as a probability, specifically:

$\alpha=P($reject $H_0 | H_0 $is true$)$.

In this case, I will have:

$P(|W| > |w| \ |H_0 $ is true$) \ < P($reject $H_0 | H_0 $is true$)\Rightarrow$ reject $H_0$

Therefore I'm saying that, if $H_0$ is true, the probability of $w$ is less than the probability of rejecting $H_0$, therefore reject $H_0$. Since, for example, I obtained $w$, its probability is lower than the probability of in rejecting $H_0$, so reject. In other words, it is more probable that I reject when I shouldn't, respect to the fact that $w$ has been sampled, so reject. This seems to me that it is more probable that I'm making a reject when I shouldn't, or there is something that escapes me. Furthermore, I'm comparing two different distributions, i.e. $P(W)$ and $P(\text{ reject } H_0)$, that seems to me two different objects, therefore I don't understand the point of comparing them.

This question can be viewed similar to The rationale behind the "fail to reject the null" jargon in hypothesis testing?, but in my opinion is different (it regards also the type II error, which is not the subject of my question) and also to What is the meaning of p values and t values in statistical tests? (which see $\alpha$ simply as a threshold, and not as a probability): Maybe a similiar question is intuition/logic behind comparing p-value and significance level but the given answer is not fully satisfactory for me, and the question has been closed since considered too similar to the other questions.

So, what is the logical meaning in rejecting the null-hypothesis when $\alpha$ is viewed as the probability to make a Type I mistake?

$\endgroup$

1 Answer 1

7
$\begingroup$

Until you study Bayesian inference this will not make as much sense to you. But you started off incorrectly. With continuous data the probability of achieving any one value of a test statistics (or of a summary measure the test statistic is based on) is zero. So the p-value is the probability of getting a test statistic that is more extreme than the observed one if $H_0$ is true and the model is correct. To read more about the enormous difference between $\alpha$ and decision errors, read this.

$\alpha$ isn't the probability of making a mistake. This is one of the most common interpretation errors and is at the root of a lot of unclear thinking by a lot of practitioners. The term "type I error rate" is a misnomer that started us off on the wrong foot a century ago. It's not a rate and is not an error probability. That's why I've moved to the term "type I assertion probability $\alpha$ in its stead. It's just an assertion trigger probability. No conditional probability that assumes $H_0$ is true can inform you about the veracity of $H_0$.

With Bayesian posterior probabilities you compute probabilities of any assertion you have and use direct reasoning. If you decide to act as if an effect is positive when P(effect > 0 | data, prior) = 0.98 you automatically get a decision error probability to carry along, which is simply 0.02.

$\endgroup$
9
  • 2
    $\begingroup$ I'm not a Bayesian and have only studied it a little, but your answer makes good sense to me. Maybe the little that I've studied it is enough. $\endgroup$
    – Peter Flom
    Commented Aug 16, 2023 at 11:34
  • 1
    $\begingroup$ It's generally stated as $\Pr(W > w)$ but you can do it either way if you want a one-sided test. But most tests are two-sided so use something like $\Pr(|W| > |w|)$. $\endgroup$ Commented Aug 16, 2023 at 12:21
  • 1
    $\begingroup$ thank you. I tried to correct my question accordly, but still there is something missing in all the reasoning. More precisely, I don't understand what you mean with "assertion probability" $\endgroup$
    – volperossa
    Commented Aug 16, 2023 at 12:56
  • 2
    $\begingroup$ In simplest terms one triggers an assertion of an effect (rejection of the supposition of no effect) when for example p < 0.05. The probability of making such an assertion under $H_0$ is 0.05 if p-values are accurately computed and the data model holds. $\endgroup$ Commented Aug 16, 2023 at 14:00
  • 3
    $\begingroup$ I have a semantic issue here. If in mathematical symbols we write $$\alpha=P({\rm reject }\, H_0 | H_0\, {\rm is\, true})$$, then is it not grammatically correct to translate this in words as "the probability of rejecting $H_0$ given that $H_0$ is true", which is an example of "making a mistake"? $\endgroup$ Commented Aug 17, 2023 at 1:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.