Why is the p-value defined the way it is (as opposed to a more intuitive measure)?

Question

As I understand it the p-value for some hypothesis H and some test-statistic T is defined as the probability of observing a test-statistic T' which is at least as extreme as T, conditional on H being true.

First question: should "at least as extreme" always be interpreted as "the absolute value of T' is at least as large as T"? If not, when shouldn't it be interpreted that way?

Second question: What is the reasoning behind defining the p-value this way as opposed to just the likelihood of the data -- "probability of observing test-statistic T, or a value in the epsilon region around T, conditional on H being true "-- which seems more clearly related to the strength of evidence that T provides with respect to H than the p-value?

Answer 1

In case you're asking for intuition, rather than for mathematical detail...

For your second question: I find it helpful to interpret the p-value as a percentile, not a probability. A p-value of 0.02 means "If $H_0$ were true, the test statistic value $T$ that I observed would have been among the top 2% possible values of $T$ that are least like $H_0$ and most like $H_A$."

For your first question: It depends on your alternative hypothesis. If you're testing whether a new drug does better than control, then perhaps you've got a one-sided $H_A$, such as "the difference in means between the treatment group vs the control group is a positive difference." If so, then you'd only reject $H_0$ if your test statistic actually points in that direction. You wouldn't want to take the absolute value of $T$ because it would not match your scientific question -- you're not interested in finding drugs that do worse than control. Again, you want to know: "Out of all the possible values of the test statistic when $H_0$ is true, is this among the top X% of values that look most like $H_A$ and least like $H_0$?"

Answer 2

First question: should "at least as extreme" always be interpreted as "the absolute value of T' is at least as large as T"? If not, when shouldn't it be interpreted that way?

The definition of the $p$-value depends on the rejection region of the test statistic. Indeed, given a sample $X_1,\ldots,X_n$, a test statistic $T(X_1,\ldots,X_n)$ and $R_\alpha$, a rejection region of size $\alpha$, then

$$ p\text{-value} = \inf\{\alpha: T(X_1,\ldots,X_n)\in R_\alpha\}. $$

Thus the $p$-value can be interpreted as the smallest size at which we can reject $H_0$. Thus the $p$-value tells us how surprising is the value of the statistic when $H_0$ is true, to be interpreted as:

the lower the $p$-value, the more surprising is observed such a value under the model with $H_0$ being true.

Indeed, some authors refer to the $p$-value by the name observed significance level.

Now about the computation of the $p$-value, if $t_n$ is the observed test statistics then:

• if the rejection region is of the form $$\{X_1,\ldots,X_n: T(X_1,\ldots,X_n)\geq c\},$$ then the $p$-value is defined by $$\sup_{\theta\in\Theta_0} P_\theta(T(X_1,\ldots,X_n)\geq t_n);$$

• if the rejection region is of the form $$\{X_1,\ldots,X_n: T(X_1,\ldots,X_n)\leq c\},$$ then the $p$-value is defined by $$\sup_{\theta\in\Theta_0} P_\theta(T(X_1,\ldots,X_n)\leq t_n);$$

• if the rejection region is of the form $$\{X_1,\ldots,X_n: T(X_1,\ldots,X_n)\geq c_1\}\cup \{X_1,\ldots,X_n: T(X_1,\ldots,X_n)\geq c_2\},$$ then the $p$-value is defined by $$2\min\left(\sup_{\theta\in\Theta_0} P_\theta(T(X_1,\ldots,X_n)\leq t_n,\sup_{\theta\in\Theta_0} P_\theta(T(X_1,\ldots,X_n)\geq t_n\right).$$

Answer 3

How is $p$-value actually defined?

Definition $1.$ (cf. $\rm[I]$) A $p$-value is a test statistic $p:\mathcal X\to [0,1]$ such that $$\mathbb P_\theta (p(\mathbf X) \leq \alpha)\leq\alpha,~~\forall\theta\in\Omega_\mathcal H, ~\forall\alpha\in(0,1).\tag{1.a}\label 1$$

Consider a series of nested tests $\langle \varphi_\alpha\rangle$ in the sense that $\varphi_\alpha(x) \leq\varphi_{\alpha^\prime}(x) $ for $\alpha<\alpha^\prime.$

Define $$\hat p:=\inf\{\alpha:\varphi_\alpha=1\}.\tag{1.b}\label b$$

Observation $1.1.$ $\hat p$ is a valid $p$-value.

Formally, if for a set of nested test functions $$\sup_{\theta\in\Omega_\mathcal H}\mathbb P_\theta(\varphi_\alpha(\mathbf X) \leq \alpha) \leq \alpha ~~\forall\alpha\in(0, 1), \tag 2\label 2$$

then for all $u\in(0,1),$ $$\mathbb P_\theta\left(\hat p\leq u\right) \leq u. \tag 3\label 3$$

$\eqref 3$ is easy to see (cf. $\rm [II]$ ) for $\left\{\hat p\leq u\right\}$ means $\{\varphi_v(\mathbf X) =1\}$ for all $u<v.$ Then, let $v\to u. $

$\blacksquare$

Now consider a test statistic $W(\mathbf X) $ whose large values indicate the rejection of $\mathcal H. $

Observation $1.2.$ (cf.$\rm [I]$) Define

$$p(\mathbf x) := \sup_{\theta\in\Omega_\mathcal H} \mathbb P_\theta(W(\mathbf X) \geq W(\mathbf x)).\tag{1.c}\label c$$ $p(\mathbf x) $ is also a valid $p$-value.

Notice that

\begin{align}p_\theta(\mathbf x) &= \mathbb P_\theta (W(\mathbf X) \geq W(\mathbf x))\\&= \mathrm F_\theta(-W(\mathbf x)),\tag 4\end{align}

which implies $p_\theta(\mathbf x) $ is stochastically greater than or equal to $\mathcal U(0, 1).$ Then as $p(\mathbf x) \geq p_\theta(\mathbf x), $ $\eqref 1$ follows.

$\blacksquare$

---

When one talks about $p$-value, they are basically meaning $\eqref 1$ or $\eqref c$ which as outlined above are genuine $p$-values.

The phrase as extreme as is essential to define $p$-value. Gloss over the definition and its equivalence as outlined above.

However, what does it imply intuitively? What does lower $p$-value mean? Why is the phrase necessary?

Over the years, there have been many CV posts dealing with the specifics. Please have a look at some of those:

$\bullet$ Why is smaller the p-value, larger is the significance?

$\bullet$ Does p-value ever depend on the alternative? (courtesy Richard Hardy)

$\bullet$ What is the meaning of p values and t values in statistical tests?

and links therein.

---

References:

$\rm [I]$ Statistical Inference, George Casella, Roger L. Berger, Wadsworth, $2002, $ sec. $8.3, $ pp. $397-398.$

$\rm [II]$ Testing Statistical Hypotheses, E. L. Lehmann, Joseph P. Romano, Springer Science$+$Business Media, $2005, $ sec. $3.3, $ pp. $63-64.$

Answer 4

Regarding the first question, you have got some good answers already. (You may want to check out the following threads, too, as I think they align with your thinking quite well: $p$-value: Fisherian vs. contemporary frequentist definitions, Does $p$-value ever depend on the alternative? and Defining extremeness of test statistic and defining $p$-value for a two-sided test).

Regarding the second question, you start with a Fisherian-esque interpretation by only looking at the null hypothesis and ignoring the alternative. This is pretty intuitive to me, but it does not seem to be fashionable anymore. However, you differ by suggesting to only look at the density at or around the test statistic – not the integral of all densities that are at most as high. The latter would be the Fisherian $p$-value*. It has the advantage of using a unified (and I think reasonably intuitive) scale between 0 and 1 which is based on ranking all possible test statistics from the most extreme (in the Fisherian sense, i.e. having the lowest likelihood*) to the least extreme (having the highest likelihood). Meanwhile, your approach would use a different scale for different tests (Student-$t$, $\chi^2$, $F$, ...), so we would have to develop intuition for each of them. While we can now quite easily judge a value between 0 and 1, there we would be dealing with all kinds of values on quite different scales – not quite as easy.

*Not everyone may agree on that; I had some discussion about it somewhere else on this site. Too bad Fisher is not here with us anymore to elaborate on his position.

Some further references:

• Keuzenkamp & Magnus "On Tests and Significance in Econometrics" (1995)

• Lehmann "The Fisher, Neyman-Pearson Theories of Testing Hypotheses: One Theory or Two?" (1993)

• Christensen "Testing Fisher, Neyman, Pearson, and Bayes" (2005)

• Spanos "Probability Theory and Statistical Inference: Econometric Modeling with Observational Data" (1999) Section 14.5; there is a newer edition, too.

Answer 5

Likelihood is a pain to work with.

For starters, likelihood can only go down as you collect more data. Indeed, if your alternate hypothesis is "My coin is biased in favour of tails" and you flip it 0 times, the likelihood of getting zero heads is 100%! Actually doing the experiment can only ever hurt your likelihood value from there.

Second, likelihood is non-exclusive. The likelihood of zero heads from zero throws is also 100% under the null, as well as under the hypothesis "It is a magic coin that always lands how I want." Even if we actually flip our coin and get tails, we're scoring a 75% likelihood. But the null hypothesis still scores a 50% likelihood. It isn't a straightforward battle between null and alternate.

Lastly, likelihood calculations are incredibly sensitive to the formulation of the hypothesis. In practice you often don't know ahead of time (and stating your hypothesis ahead of time is important!) what exact numbers to put in. If you suspect someone is cheating with a coin biased for tails, it's a pain to have to say whether it's a 70% tails or a 75% tails coin. Now suppose you take that coin and flip it 1000 times, getting exactly 748 tails. Under the 75% hypothesis your likelihood of that result is an astounding 2.9%, but under a 70% hypothesis it would be 0.01%. Under the hypothesis we really want, "It's biased to somewhere in the 65-80 zone" it's just messy to define and calculate.

The intuitive measure you're after is "Probability of our hypothesis." Unfortunately, it generally is not a value we can actually calculate from our experiment. It depends too much on things like priors (Fair coins are more common than magic coins, but how much?), and the huge space of possibility of other hypotheses (e.g. what if the coin is fair but the flipper is trained?).

So, we make do with p values as our first filter. To be clear, a tiny p value does not mean that the alternate hypothesis is true. All it means is that the researcher has done enough work to beat the trivial "It's a fluke" standard that they've earned the right to have their work examined.