Confused about rejection region and P-value

Question

I am confused about the rejection region and P-value. I thought that the P-value is the simply the probability associated to the set of points where we would reject the null hypothesis (the rejection region). But according to this response, they are not related. Is it then possible, having $R$ as a test statstic, to have a rejection region, say $C=\{R^{observed}:R^{observed}\geq w_{1−\frac{\alpha}{2}}\}\cup\{R^{observed}:R^{observed}\leq w_{\frac{\alpha}{2}}\}$, but the p-value is $P(R>R^{observed})$?

Answer 1

I think this will be best understood with an example. Let us solve a hypothesis test for the mean height of people in a country. We have the information about the heights of a sample of people in that country. First, we define our null and alternative hypthesis:

• $H_0: \mu \geq a$
• $H_1: \mu < a$

And (let me change your notation) we have our test statistic $Z$. Now, we know two things about this test statistic:

1. We know the formula for this statistic: $Z=\dfrac{\bar{x}-\mu}{\sigma/\sqrt{n}}$

2. We know the distribution it follows. For simplicity in this explanation, let me assume it follows a normal distribution. We would have that $Z\sim N(0,1)$.

Now the important part: We do not know the true population value of $\mu$ (this means, we do not know the true mean height of all the people in the country, if we wanted to know that we would need to know the height of all the citizens in that country). But we can say: hey, let's assume that the value for $\mu$ is the value stated in $H_0$ (a.k.a. let's assume that $\mu=a$)

And we pose the key question: How likely is it for $\mu$ to take the value $a$ given the information that we know from the sample data?

Now that we are assuming that $\mu=a$, we can obtain the value of our test statistic under the null hypothesis (this is, assuming that $\mu=a$).

There can be two possible results:

• If $a$ is not a likely value for $\mu$ to have then the statistic $\hat{Z}$ value will not fit well in the distribution $Z$ follows and we will reject $H_0$

• If $a$ is a likely value for $\mu$ to have, then the statistic $\hat{Z}$ value will fit well in the distribution $Z$ follows and we will fail to reject $H_0$:

And finally here comes into play the rejection region and the p-value.

• We will consider that the tail of the distribution (in this case, the left tail, as stated by $H_1$ are all not likely values for $\hat{Z}$ so if any value is close to the tail, we reject $H_0$. How close to the tail? That is stated by the significance level $\alpha$. The rejection region is:

$$RR=\{Z {\ }s.t.{\ } Z < -Z_{\alpha}\}$$

If we take, for example, $\alpha=0.05$ then the rejection region is $$RR=\{Z {\ }s.t.{\ } Z < -Z_{0.05}\}= \{Z {\ }s.t.{\ } Z < -1.645\}$$

• And the p-value is simply the probability of obtaining a value at least as extreme as the one from our sample, or in other words, if the sample value of our statistic is $\hat{Z}$ then the p-value is $$p-value=P(Z<\hat{Z})$$

In one image, in red the rejection region, and in green the p-value.

Remark: This plots have been made assuming that we are doing a left sided test. Considering a right sided or two sided test would yield similar but not equal images.

Answer 2

The rejection region is fixed beforehand. If the null hypothesis is true then some $\alpha \%$ of the observations will be in the region.

The p-value is not the same as this $\alpha \%$.

The p-value is computed for each separate observation, and can be different for two observations that both fall inside the rejection region.

The p-value indicates how extreme* a value is. And expresses this in terms of a probability. This expression in terms of a probability could be seen as the quantile of the outcome when the potential outcomes are ranked in decreasing order of extremity. The more extreme the observation, the lower the quantile.

In short: The rejection region can be seen as the region of observations for which the associated quantile or p-value is lower than some value.

See also: https://stats.stackexchange.com/questions/tagged/critical-value

---

^{* What is and what is not considered extreme is not well defined here and might be considered arbitrary, but depending on the situation there might be good reasons to choose a particular definition. For example, think about one-sided and two-sided tests in which case different sorts of extremities are chosen.}

^{Because of the variations in choice for 'extremeness', it might be that you encounter a situation where some observation is inside the rejection region but has a p-value that is larger. This is the case when the two use a different definition. But typically the p-value and rejection region should relate to the same definition of 'extremeness'.}