2
$\begingroup$

Could you provide formal definition for p-value? Or, do you have any good source for it?

For a day, I've been searching for the formal definition of p-value, but I couldn't yet. The majority of the book of statistics are mathematically non-rigorous and most of them didn't define p-value formally. I found two books below that are mathematically rigorous, but the definition by these two are also ambiguous.

For example, in p127 "Mathematical Statistics, 2nd edition" by Jun Shao:

It is good practice to determine not only whether $H_0$ is rejected or accepted for a given $\alpha$ and a chosen test $T_\alpha$, but also the smallest possible level of significance at which $H_0$ would be rejected for the computed $T_{\alpha}(x)$, i.e. $\hat{\alpha} = \mathrm{inf}\left\{ \alpha \in (0, 1) : T_\alpha(x) = 1 \right\}$. Such an $\hat{\alpha}$, which depends on $x$ and the chosen test and is a static, is called the p-value for the test $T_{\alpha}$.

However, In this book, $T_{\alpha}$ is not defined before the above statement.

In other book, p63, "Testing Statistical Hypothesis 3rd edition" by E.L. Lehmann:

... When this is the case, it is good practice to determine not only whether the hypothesis is accepted or rejected at the given significance level, but also to determine the smallest significance level, or more formally

$$ \hat{p} = \hat{p}(X) = \mathrm{inf}\left\{ \alpha : X \in S_{\alpha} \right\} $$

at which the hypothesis would be rejected for the given observation. This number, the so-called p-value given an idear of how strongly the data contradicts the hypothesis.

But unfortunately I couldn't find the definition of $S_{\alpha}$ so the same situation as Shao's book.

$\endgroup$
2
  • $\begingroup$ See also here - note the comments that point out to some intricacies missed in basic stat textbooks. $\endgroup$
    – Roger V.
    Mar 16 at 8:39
  • $\begingroup$ Honestly, I don't see either of the links as giving a full definition of the p value. What I am missing is a clear definition of "rejection" or sometimes "more extreme", as well as the crucial assumption that the model is correctly specified (which it never is). Somewhat surprisingly, we don't seem to have a thread carrying both the "statistical-significance" and the "references" tag that is useful. $\endgroup$ Mar 16 at 9:26

2 Answers 2

6
$\begingroup$

Lehmann is talking about a nested sequence of critical regions $\langle S_\alpha\rangle$ with the index being the size of the corresponding test. This is due to the fact that he needs to find the smallest significance level.

Shao is also using the same concept; however instead of defining the $p$-value in terms of $S_\alpha, $ the author used the critical function (using Lehmann terminology) in that $T_\alpha(\mathbf x) =1\implies \mathbf x\in S_\alpha.$

Note both are talking about non-randomized test procedures.

Generalizing to the randomized case is not difficult either. Lehmann explains further that in that case, one can resort to the nested tests $\langle \varphi_\alpha\rangle.$

References have been provided in my answer here; see Ben's post for how the nested argument emanated from imposing a certain evidentiary order relation. For a brief philosophical take, check my post here.

$\endgroup$
4
  • $\begingroup$ +1 - couldn't the implication be an equivalence? Or are there subtleties relating to, e.g., randomized tests? $\endgroup$ Mar 16 at 9:11
  • $\begingroup$ @ChristophHanck, are you saying $T_\alpha(\mathbf x) = 1\iff \mathbf x\in S_\alpha$? $\endgroup$ Mar 16 at 15:35
  • $\begingroup$ That is what I am conjecturing, yes, without however having seen the definitions in the original sources. $T=1$ sounds like "test rejects" to me and $x\in S$ like "sample is such that we are in the rejection region", which seems equivalent. $\endgroup$ Mar 16 at 16:23
  • 1
    $\begingroup$ Exactly, that would be affirmative @ChristophHanck. $\endgroup$ Mar 16 at 16:25
4
$\begingroup$

But unfortunately I couldn't find the definition of $S_{\alpha}$

The p-value is not defined in an unambiguous way

"The probability to get, given the null hypothesis, an effect-size equal to or larger than the observed effect-size"

The culprit is that 'effect-size' is not unambiguously defined. It depends on arbitrary choices.

For the same experiment different methods can compute different p-values (depending on different methods to define $S_\alpha$, different definitions of effect size).

So the p-value has indirectly no unambiguous formal definition. It is also more something like a concept rather than a rigorous mathematical construction. Statistics is more than objective mathematical formulas.

For more specific cases like a particular hypothesis test, the p-value can be expressed in an unambiguous formal way. E.g. for a one sided z-test with $H_0: \mu \leq 0$ and $H_a: \mu > 0$ the p-value is defined as $p = 1-\Phi(z)$ where $\Phi$ is the cumulative distribution function of the standard normal distribution.


may I ask you if you can expand the point For the same experiment different methods can compute different p-values?

For many observations with only a single dimension, there is often a natural order and most approaches agree, and if they disagree then it is not because they have a different view of effect-size, but because the methods might be approximations of the actual p-value and are not exact methods to compute the p-value. Yet, a typical difference is the difference between one-sided and two-sided tests (example: Why does $\mu > 0$ (or even $\mu > \epsilon$) "seem easier” to substantiate than $\mu \neq 0$?)

When the data is multivariate then what is and what is not extreme becomes even more ambiguous than just the difference between one-sided and two-sided. One has to draw regions. An example of a difference occurs here:

$\endgroup$
3
  • $\begingroup$ +1. Nicely articulated. But may I ask you if you can expand the point For the same experiment different methods can compute different p-values? $\endgroup$ Mar 16 at 9:02
  • $\begingroup$ @User1865345 I have added this to the answer $\endgroup$ Mar 16 at 9:51
  • $\begingroup$ Thanks for the extensive addition to the answer. $\endgroup$ Mar 16 at 10:57

Not the answer you're looking for? Browse other questions tagged or ask your own question.