3
$\begingroup$

Why do we also consider the probability of more extreme events in hypothesis testing?

I'm a beginner in statistics and I am trying to understand why do we take into consideration the probability of more extreme events when doing a hypothesis test. Let's take an example so I can explain exactly what I mean.

Let's say that we have the $2$ hypothesis:

$$H_0: \mu = 780$$

$$H_a: \mu \ne 780$$

And we set the significance level as $\alpha = 0.05$.

Now let's say we take a sample of $n = 25$ and we find that the mean of this sample is $\bar{X} = 776$. Assume that we (somehow) know the standard deviation of the population to be $\sigma = 16$.

What I understood from what I've read so far is that we assume that the null hypothesis is true. So that means that the random variable

$$\frac{\bar{X} - \mu}{\sigma_{\bar{X}}}$$

where

$$\sigma_{\bar{X}} = \frac{\sigma}{\sqrt{n}}$$

is actually the standard normal, with a mean of $0$ and a standard deviation of $1$. So out of this we calculate the $z$ statistic with the information given in the problem and get the value (after some calculations):

$$z = -1.25$$

So far, so good. Now we need to calculate the p-value. And to do so we add the probability of having $Z < -1.25$ and of having $Z > 1.25$, where $Z$ is the standard normal. This is what I don't understand. Why do we take into consideration all of that area:

enter image description here

Wouldn't that overestimate the p-value? I understand that we cannot consider just the observed value, but why don't we just use something like a small interval around the observed value? Why do we consider all of the events that are more extreme than the observed value? And we also consider the event that is equally likely to happen, namely $z = 1.25$, and after that we also consider the events that are more extreme than $z=1.25$ too. This is also a mystery to me. Don't we want to find the probability of that event happening, so that after finding this probability we can decide if we reject the null hypothesis or not? So if we want to find the probability of the observed value happening (given that the null hypothesis is true), then why do we also add to that the probabilities of all the events that are equally likely to happen and then all of the probabilities of the events that are less likely to happen?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ (+1) This is exactly the question I set out to answer in the duplicate at stats.stackexchange.com/a/130772/919. $\endgroup$ – whuber♦ Apr 7 '20 at 23:44
  • 1
    $\begingroup$ For a brief answer, I think this is partly historical. The Bayesian view of hypothesis testing is more natural, and is what you want. Unfortunately, it is undermined by the fact there's no single "best" choice of prior in every case. Hence, historically, the was a much greater focus on frequentist statistics, which, unfortunately, is somewhat hard to interpret. $\endgroup$ – Tim Mak Apr 8 '20 at 4:02
  • 1
    $\begingroup$ @Bogdan I think it's important to begin first by thinking in terms of a rejection region (initially in general terms -- later on in the process, specific boundary value(s) of the rejection region will be defined by your significance level); that is, to consider what sorts of values of your test statistic you'll consider as being most consistent with your alternative (in a specific sense, "far from" your null). From that, almost everything else follows quite simply. I'd also recommend Scortchi's answer here: stats.stackexchange.com/questions/44769/understanding-p-value $\endgroup$ – Glen_b Apr 8 '20 at 6:55