# How to define the region that mathematically defines the p-value?

This is a question concerning the definition of the p-value in a more general than ordinary case.

Let $$X\sim f(x;\theta)$$ be a population and $$(X_i)_{i=1}^n$$ a random sample (i.i.d. $$\sim f(\cdot;\theta)$$, by definition). Let $$(\mathcal X, \mathcal B)$$ be the sample space (if $$X\sim f(\cdot;\theta)$$ takes real values, $$\mathcal X = \mathbb R^n$$).

Suppose $$\Theta$$ is the parameter space. Remember that any $$\theta \in \Theta$$ defines a probability $$P_\theta$$ on $$(\mathcal X, \mathcal B)$$. So, for any $$\mathcal{C}\in \mathcal B$$, we can calculate $$P_\theta (\mathcal C)$$.

I have the following hypotheses about $$\theta$$:

$$H_0: \theta \in \Theta_0, \quad \text{vs} \quad H_1: \theta \notin \Theta_0$$

A test is a decision rule in the sense that $$d:\mathcal X \to \{\text{Accept } H_0, \text{Reject } H_0\}$$. A test is characterized by a test statistic $$T:\mathcal X \to \mathbb R$$. For this, we define the critical region as $$\mathcal C =\{x \in \mathcal X : d(x)= \text{Reject } H_0 \}= \{ x \in \mathcal X : T(x) \in R\}$$ Here, $$R$$ is the rejection region.

Suppose the realization $$\bar x \in \mathcal X$$ or $$\bar t = T(\bar x)$$. We know that the p-value is the probability of obtaining test results at least as extreme as the result actually observed, under the assumption that the null hypothesis is correct.

But how do we define the region $$R_{extr}$$ corresponding to "at least as extreme as the result actually observed"?

In some cases, the region $$R_{extr}$$ is trivial: $$p_v=P_\theta(R_{extr})=P_\theta( |T| \geq |\bar t| ), \quad \theta = \theta_0\quad (\text{i.e. } \Theta_0 =\{\theta_0\})$$ where $$T$$ is the normal or t-student distribution.

However, I have a preference for defining more general cases and then studying the particular cases. Could you help me?

• You start with the decision rule. The p-value is the largest $\alpha$ for which the critical region of level $\alpha$ contains the sample statistic. Many decision rules are determined by the likelihood ratio as I explain at stats.stackexchange.com/a/130772/919. Perhaps that post (or something else in that thread) answers your question?
– whuber
Jun 21 at 20:19
• I think that it is a mistake to attempt to define p-values in the context of acceptance or rejection regions, even if it might be possible or popular to do so. The p-value was around well before the Neyman–Pearsohnian paradigm of rejection regions was invented and their invention did not utilise p-values at all. P-values have a life outside of the confines of acceptance and rejection regions. Jun 21 at 20:44

A p-value is determined by combination of the observed data and a chosen statistical model. It does not require a decision rule, a critical value, or a rejection region.

The model will include a statistic distilled from the data that can aligned with to the theoretical distribution of such a statistic from that statistical model. To get a p-value you compare the observed value of the statistic with the theoretical distribution of statistic values that are obtained when the null hypothesis is set to 'true'. Given that the null hypothesis usually corresponds to a particular parameter value within the statistical model, setting it to 'true' is achieved by setting that parameter to that value.

Given that setup it is easy to define 'extremeness'. Simply rank the statistic values expected by the model from small to large and the extreme values will have low ranks or high ranks. A one-tailed p-value is the fractional rank of the observed value of the statistic among all values of the statistic expected from the statistical model when the null hypothesis is set to true.

(I suspect that you would prefer this to be explained and defined with your lovely equations and symbols. That is beyond me and so I will use the trick of textbook writers and say that I am leaving that up to the reader as an exercise!)

• The problem with this approach is that it is conceptually arbitrary. Why should values of a statistic in the tails of its null distribution be of interest? Why not rank the values in some other order than absolute value? That turns out to be the correct approach in the example I gave in the link, for instance. Another example of this is Fisher's examination of Mendel's pea results, which followed the null hypothesis too closely.
– whuber
Jun 21 at 22:55
• @Whuber I don't agree that it is arbitrary. You can choose a different statistic if you want the p-value to reflect some different property of the data, but if you want the p-value to behave as a p-value you will need to effectively rank the statistic values from the model with the null set to true. Jun 22 at 1:09
• @whuber When you say 'absolute value' are you thinking of a two-tailed p-value? For one-tailed p-values (which I prefer) the test statistics are not converted to the absolute values (i.e. negatives are made positive), but are ranked on their actual value. Jun 22 at 1:11
• @whuber When you say "The problem with this approach" are you referring to my explanation of p-values or to the use of p-values? Jun 22 at 1:12
• (1) "Effectively rank" is what makes the approach arbitrary. (2) I used absolute value to reflect your example that refers to "small" and "large" values. (3) "Approach" refers to your post.
– whuber
Jun 22 at 13:02