How to define the region that mathematically defines the p-value?

Question

This is a question concerning the definition of the p-value in a more general than ordinary case.

Let $X\sim f(x;\theta)$ be a population and $(X_i)_{i=1}^n$ a random sample (i.i.d. $\sim f(\cdot;\theta)$, by definition). Let $(\mathcal X, \mathcal B)$ be the sample space (if $X\sim f(\cdot;\theta)$ takes real values, $\mathcal X = \mathbb R^n$).

Suppose $\Theta$ is the parameter space. Remember that any $\theta \in \Theta$ defines a probability $P_\theta$ on $(\mathcal X, \mathcal B)$. So, for any $\mathcal{C}\in \mathcal B$, we can calculate $P_\theta (\mathcal C)$.

I have the following hypotheses about $\theta$:

$$H_0: \theta \in \Theta_0, \quad \text{vs} \quad H_1: \theta \notin \Theta_0$$

A test is a decision rule in the sense that $d:\mathcal X \to \{\text{Accept } H_0, \text{Reject } H_0\}$. A test is characterized by a test statistic $T:\mathcal X \to \mathbb R$. For this, we define the critical region as $$\mathcal C =\{x \in \mathcal X : d(x)= \text{Reject } H_0 \}= \{ x \in \mathcal X : T(x) \in R\}$$ Here, $R$ is the rejection region.

Suppose the realization $\bar x \in \mathcal X$ or $\bar t = T(\bar x)$. We know that the p-value is the probability of obtaining test results at least as extreme as the result actually observed, under the assumption that the null hypothesis is correct.

But how do we define the region $R_{extr}$ corresponding to "at least as extreme as the result actually observed"?

In some cases, the region $R_{extr}$ is trivial: $$p_v=P_\theta(R_{extr})=P_\theta( |T| \geq |\bar t| ), \quad \theta = \theta_0\quad (\text{i.e. } \Theta_0 =\{\theta_0\})$$ where $T$ is the normal or t-student distribution.

However, I have a preference for defining more general cases and then studying the particular cases. Could you help me?

Answer 1

A p-value is determined by combination of the observed data and a chosen statistical model. It does not require a decision rule, a critical value, or a rejection region.

The model will include a statistic distilled from the data that can aligned with to the theoretical distribution of such a statistic from that statistical model. To get a p-value you compare the observed value of the statistic with the theoretical distribution of statistic values that are obtained when the null hypothesis is set to 'true'. Given that the null hypothesis usually corresponds to a particular parameter value within the statistical model, setting it to 'true' is achieved by setting that parameter to that value.

Given that setup it is easy to define 'extremeness'. Simply rank the statistic values expected by the model from small to large and the extreme values will have low ranks or high ranks. A one-tailed p-value is the fractional rank of the observed value of the statistic among all values of the statistic expected from the statistical model when the null hypothesis is set to true.

(I suspect that you would prefer this to be explained and defined with your lovely equations and symbols. That is beyond me and so I will use the trick of textbook writers and say that I am leaving that up to the reader as an exercise!)