Confused about rejection region and P-value

I am confused about the rejection region and P-value. I thought that the P-value is the simply the probability associated to the set of points where we would reject the null hypothesis (the rejection region). But according to this response, they are not related. Is it then possible, having $$R$$ as a test statstic, to have a rejection region, say $$C=\{R^{observed}:R^{observed}\geq w_{1−\frac{\alpha}{2}}\}\cup\{R^{observed}:R^{observed}\leq w_{\frac{\alpha}{2}}\}$$, but the p-value is $$P(R>R^{observed})$$?

I think this will be best understood with an example. Let us solve a hypothesis test for the mean height of people in a country. We have the information about the heights of a sample of people in that country. First, we define our null and alternative hypthesis:

• $$H_0: \mu \geq a$$
• $$H_1: \mu < a$$

And (let me change your notation) we have our test statistic $$Z$$. Now, we know two things about this test statistic:

1. We know the formula for this statistic: $$Z=\dfrac{\bar{x}-\mu}{\sigma/\sqrt{n}}$$

2. We know the distribution it follows. For simplicity in this explanation, let me assume it follows a normal distribution. We would have that $$Z\sim N(0,1)$$.

Now the important part: We do not know the true population value of $$\mu$$ (this means, we do not know the true mean height of all the people in the country, if we wanted to know that we would need to know the height of all the citizens in that country). But we can say: hey, let's assume that the value for $$\mu$$ is the value stated in $$H_0$$ (a.k.a. let's assume that $$\mu=a$$)

And we pose the key question: How likely is it for $$\mu$$ to take the value $$a$$ given the information that we know from the sample data?

Now that we are assuming that $$\mu=a$$, we can obtain the value of our test statistic under the null hypothesis (this is, assuming that $$\mu=a$$).

There can be two possible results:

• If $$a$$ is not a likely value for $$\mu$$ to have then the statistic $$\hat{Z}$$ value will not fit well in the distribution $$Z$$ follows and we will reject $$H_0$$

• If $$a$$ is a likely value for $$\mu$$ to have, then the statistic $$\hat{Z}$$ value will fit well in the distribution $$Z$$ follows and we will fail to reject $$H_0$$:

And finally here comes into play the rejection region and the p-value.

• We will consider that the tail of the distribution (in this case, the left tail, as stated by $$H_1$$ are all not likely values for $$\hat{Z}$$ so if any value is close to the tail, we reject $$H_0$$. How close to the tail? That is stated by the significance level $$\alpha$$. The rejection region is:

$$RR=\{Z {\ }s.t.{\ } Z < -Z_{\alpha}\}$$

If we take, for example, $$\alpha=0.05$$ then the rejection region is $$RR=\{Z {\ }s.t.{\ } Z < -Z_{0.05}\}= \{Z {\ }s.t.{\ } Z < -1.645\}$$

• And the p-value is simply the probability of obtaining a value at least as extreme as the one from our sample, or in other words, if the sample value of our statistic is $$\hat{Z}$$ then the p-value is $$p-value=P(Z<\hat{Z})$$

In one image, in red the rejection region, and in green the p-value.

Remark: This plots have been made assuming that we are doing a left sided test. Considering a right sided or two sided test would yield similar but not equal images.

• Thank you for your explanation! So If we wanted to test $H_0: \mu =a$ vs $H_1:\mu \neq a$, then $RR=\{Z s.t. Z_{1-\frac{\alpha}{2}} <Z < Z_{\frac{\alpha}{2}}\}$, and the P-value would be the probability $P(|Z|<\hat{Z})$. Right? Feb 9, 2021 at 10:35
• Yeah, that is right. Feb 9, 2021 at 10:52
• Suppose then we want to test some other parameter $H_0: \theta = a$ vs $H_1: \theta \neq a$, and that our test statistic $T$ under the null hypothesis has a Gamma distribution $\Gamma(m,n)$. The rejection region would still the same except that the quantile changes to that of a $\Gamma(m,n)$. What I'm having trouble with is : what is the P-value in this case? Feb 9, 2021 at 10:58
• Well in two sided tests as you said you can always obtain the rejection region, but I am afraid that the two sided p-value is only well defined when the test statistic has a symetric distribution. Feb 9, 2021 at 11:08

The rejection region is fixed beforehand. If the null hypothesis is true then some $$\alpha \%$$ of the observations will be in the region.

The p-value is not the same as this $$\alpha \%$$.

The p-value is computed for each separate observation, and can be different for two observations that both fall inside the rejection region.

The p-value indicates how extreme* a value is. And expresses this in terms of a probability. This expression in terms of a probability could be seen as the quantile of the outcome when the potential outcomes are ranked in decreasing order of extremity. The more extreme the observation, the lower the quantile.

In short: The rejection region can be seen as the region of observations for which the associated quantile or p-value is lower than some value.

* What is and what is not considered extreme is not well defined here and might be considered arbitrary, but depending on the situation there might be good reasons to choose a particular definition. For example, think about one-sided and two-sided tests in which case different sorts of extremities are chosen.

Because of the variations in choice for 'extremeness', it might be that you encounter a situation where some observation is inside the rejection region but has a p-value that is larger. This is the case when the two use a different definition. But typically the p-value and rejection region should relate to the same definition of 'extremeness'.

• Does it mean that the rejection region can be $C=\{R^{observed}:R^{observed}\geq w_{1−\frac{\alpha}{2}}\}\cup\{R^{observed}:R^{observed}\leq w_{\frac{\alpha}{2}}\}$, but a P-value $P(R>R^{observed})$ (meaning that "extreme" is when $R>R^{observed}$)? Feb 9, 2021 at 11:02
• Ah now I see your problem. $P(R>R^{observed})$ can be very high. Say you have the hypothesis $R \sim N(0,\sigma^2)$ (ie normal distributed). The 5% rejection region could be when for the absolute value $|R|>2\sigma$. In that case, if you have an observation below $-2$, that is $R^{observed}<-2$, then the observation is inside the rejection region, but the probability to observe $R>R^{observed}$ is very high... Feb 9, 2021 at 11:15
• ... the discrepancy occurs because the probability $P(R>R^{observed})$ is not using the same definition for an extreme value as the definition that has been used for the rejection region. Feb 9, 2021 at 11:16
• @ToneyShields, this has to do with the original Fisherian understanding of $p$-value (how extreme the result is as determined by $H_0$ alone, i.e. the sampling distribution of the test statistic under $H_0$) versus the modern Fisher-Neyman-Pearson hybrid ((how extreme the result is as determined by $H_0$ and $H_1$ together). I have a few related threads here. In that (and other) regard(s), the footnote of Sextus' answer is an important one. Feb 9, 2021 at 12:41