This question already has an answer here:

As per statistical literature, there seems to be this understanding that given a p value less than alpha, we are to reject the null hypothesis.

The way I am interpreting this is that is the critical region, as defined by the alpha is the 'False Rejection' (TYPE 1 error) zone and constitutes the error threshold, why then p-value for a sample that is less than the alpha and within the error zone results in rejection of the null hypothesis?

Another line of thought - if the p-value is a way of saying how extreme a test statistic is for our sample data and seems to have a low probability, then how does it constitute evidence against the null hypothesis? Where am I going wrong?


marked as duplicate by whuber Dec 6 '18 at 19:11

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • $\begingroup$ The p-value is calculated under the null model. Hence if the test statistic that was observed or a larger value has a low probability under the null model, then it suggests that the null model is unlikely to have been the model that generated the observed data. $\endgroup$ – hard2fathom Dec 6 '18 at 18:56

The "alpha" ($\alpha$) is set a priori to be the probability of rejecting the null hypothesis when the null hypothesis is true (the Type I error as you stated). If, after conducting your statistical test, the $p$-value is found to be less than $\alpha$, this shows that, under the assumed null hypothesis, the probability you'd observe a test statistic as large (or larger) as the one you did, is very small. Something with such a small small probability of occuring should be considered suspicious. How could something be observed when it's so unlikely? Two things could have happened:

1) The assumption, under the null hypothesis is wrong (so you reject it/fail to accept it); or

2) You got a very unlucky result. This is simply the risk you are willing to take and you should set your $\alpha$/critical region accordingly when you set up your test. If you can tolerate more risk, set $\alpha$ to be larger. If you can't, set $\alpha$ to be smaller.

  • $\begingroup$ "likelihood you'd observe a test statistic as large" should be "probability you'd observe a test statistic as large". "likelihood" has a specific meaning and doesn't apply here. $\endgroup$ – A. Donda Dec 6 '18 at 19:54
  • 1
    $\begingroup$ That's a fair point, @A.Donda. I fell into the trap of interchanging everyday English with statistical terminology. The "likelihood" I mentioned above, should be taken to mean the everyday English version of it (i.e. probability). $\endgroup$ – StatsStudent Dec 6 '18 at 20:58

Not the answer you're looking for? Browse other questions tagged or ask your own question.