Formal definition of p-value [duplicate]

Question

Could you provide formal definition for p-value? Or, do you have any good source for it?

For a day, I've been searching for the formal definition of p-value, but I couldn't yet. The majority of the book of statistics are mathematically non-rigorous and most of them didn't define p-value formally. I found two books below that are mathematically rigorous, but the definition by these two are also ambiguous.

For example, in p127 "Mathematical Statistics, 2nd edition" by Jun Shao:

It is good practice to determine not only whether $H_0$ is rejected or accepted for a given $\alpha$ and a chosen test $T_\alpha$, but also the smallest possible level of significance at which $H_0$ would be rejected for the computed $T_{\alpha}(x)$, i.e. $\hat{\alpha} = \mathrm{inf}\left\{ \alpha \in (0, 1) : T_\alpha(x) = 1 \right\}$. Such an $\hat{\alpha}$, which depends on $x$ and the chosen test and is a static, is called the p-value for the test $T_{\alpha}$.

However, In this book, $T_{\alpha}$ is not defined before the above statement.

In other book, p63, "Testing Statistical Hypothesis 3rd edition" by E.L. Lehmann:

... When this is the case, it is good practice to determine not only whether the hypothesis is accepted or rejected at the given significance level, but also to determine the smallest significance level, or more formally

$$ \hat{p} = \hat{p}(X) = \mathrm{inf}\left\{ \alpha : X \in S_{\alpha} \right\} $$

at which the hypothesis would be rejected for the given observation. This number, the so-called p-value given an idear of how strongly the data contradicts the hypothesis.

But unfortunately I couldn't find the definition of $S_{\alpha}$ so the same situation as Shao's book.

Answer 1

Lehmann is talking about a nested sequence of critical regions $\langle S_\alpha\rangle$ with the index being the size of the corresponding test. This is due to the fact that he needs to find the smallest significance level.

Shao is also using the same concept; however instead of defining the $p$-value in terms of $S_\alpha, $ the author used the critical function (using Lehmann terminology) in that $T_\alpha(\mathbf x) =1\implies \mathbf x\in S_\alpha.$

Note both are talking about non-randomized test procedures.

Generalizing to the randomized case is not difficult either. Lehmann explains further that in that case, one can resort to the nested tests $\langle \varphi_\alpha\rangle.$

References have been provided in my answer here; see Ben's post for how the nested argument emanated from imposing a certain evidentiary order relation. For a brief philosophical take, check my post here.

Answer 2

But unfortunately I couldn't find the definition of $S_{\alpha}$

The p-value is not defined in an unambiguous way

"The probability to get, given the null hypothesis, an effect-size equal to or larger than the observed effect-size"

The culprit is that 'effect-size' is not unambiguously defined. It depends on arbitrary choices.

For the same experiment different methods can compute different p-values (depending on different methods to define $S_\alpha$, different definitions of effect size).

So the p-value has indirectly no unambiguous formal definition. It is also more something like a concept rather than a rigorous mathematical construction. Statistics is more than objective mathematical formulas.

For more specific cases like a particular hypothesis test, the p-value can be expressed in an unambiguous formal way. E.g. for a one sided z-test with $H_0: \mu \leq 0$ and $H_a: \mu > 0$ the p-value is defined as $p = 1-\Phi(z)$ where $\Phi$ is the cumulative distribution function of the standard normal distribution.

---

may I ask you if you can expand the point For the same experiment different methods can compute different p-values?

For many observations with only a single dimension, there is often a natural order and most approaches agree, and if they disagree then it is not because they have a different view of effect-size, but because the methods might be approximations of the actual p-value and are not exact methods to compute the p-value. Yet, a typical difference is the difference between one-sided and two-sided tests (example: Why does $\mu > 0$ (or even $\mu > \epsilon$) "seem easier” to substantiate than $\mu \neq 0$?)

When the data is multivariate then what is and what is not extreme becomes even more ambiguous than just the difference between one-sided and two-sided. One has to draw regions. An example of a difference occurs here:

• Surprising behavior of the power of Fisher exact test (permutation tests) The image shows results from two binomial distributed variables $X,Y \sim B(500,p)$ with two different tests for $H_0:p=0.5$

• R Tukey HSD Anova: Anova significant, Tukey not?

  How can I get a significant overall ANOVA but no significant pairwise differences with Tukey's procedure?

  Comparing two, or more, independent paired t-tests

  

  The image shows difference in rejection regions for the hypothesis $H_0: \mu_1=\mu_2=\mu_3$ based on the observed t-values of individual comparison between the three possible pairs.

• Which statistical analysis should I perform if the data sets are not normally distributed?

  

  This compares a Mann-Whitney U test versus t-test (which both test equality of distributions, but one does this by testing the relative dominance of distributions $P(X<Y) = P(Y>X)$ and the other by testing the equality of means $\mu_X = \mu_Y$)