# Why is the p-value defined the way it is (as opposed to a more intuitive measure)?

As I understand it the p-value for some hypothesis H and some test-statistic T is defined as the probability of observing a test-statistic T' which is at least as extreme as T, conditional on H being true.

First question: should "at least as extreme" always be interpreted as "the absolute value of T' is at least as large as T"? If not, when shouldn't it be interpreted that way?

Second question: What is the reasoning behind defining the p-value this way as opposed to just the likelihood of the data -- "probability of observing test-statistic T, or a value in the epsilon region around T, conditional on H being true "-- which seems more clearly related to the strength of evidence that T provides with respect to H than the p-value?

• The probability of exactly the same sample statistic, conditional on the usual stuff, is of measure zero whenever the sample statistic is continuous, as it usually is, so that would be useless in that circumstance. To follow, your proposal depends on choosing epsilon too! Not the answer you're seeking, necessarily, but this kind of question raises many issues, several of which imply that researchers should reach instead for confidence intervals, or indeed a pure likelihood or even Bayesian approach. (Detail: tests are often one-tailed, but your wording comes near to coping with that.) Nov 11, 2022 at 12:52
• Here are a few related threads on the topic of your first question: Defining extremeness of test statistic and defining $p$-value for a two-sided test, Does $p$-value ever depend on the alternative? and $p$-value: Fisherian vs. contemporary frequentist definitions. Nov 11, 2022 at 17:22
• Regarding your 2nd question: The logic behind the definition of p-values has been debated even among academics. See for example rejoinder 6 in Testing precise hypotheses by Berger and Delampady for some criticism of the "more extreme" part of the definition. In particular: "The inclusion of all data ’more extreme’ than the actual $x_0$ is a curious step, and one which we have seen no remotely convincing justification for." Don't expect a straight-forward answer. Nov 11, 2022 at 20:44
• A useful way to think of testing is that testing addresses the question: "Are the data (statistically) consistent with the null hypothesis"? Thus you will be less likely to misinterpret it to be be a measure of relative plausibility. When there are two precise hypotheses (a very strong assumption), we should not be surprised if the data is inconsistent with both. Note there is no need to explicitly state an alternative hypothesis (David Cox called this a pure significance test). Nov 11, 2022 at 22:07

In case you're asking for intuition, rather than for mathematical detail...

For your second question: I find it helpful to interpret the p-value as a percentile, not a probability. A p-value of 0.02 means "If $$H_0$$ were true, the test statistic value $$T$$ that I observed would have been among the top 2% possible values of $$T$$ that are least like $$H_0$$ and most like $$H_A$$."

For your first question: It depends on your alternative hypothesis. If you're testing whether a new drug does better than control, then perhaps you've got a one-sided $$H_A$$, such as "the difference in means between the treatment group vs the control group is a positive difference." If so, then you'd only reject $$H_0$$ if your test statistic actually points in that direction. You wouldn't want to take the absolute value of $$T$$ because it would not match your scientific question -- you're not interested in finding drugs that do worse than control. Again, you want to know: "Out of all the possible values of the test statistic when $$H_0$$ is true, is this among the top X% of values that look most like $$H_A$$ and least like $$H_0$$?"

First question: should "at least as extreme" always be interpreted as "the absolute value of T' is at least as large as T"? If not, when shouldn't it be interpreted that way?

The definition of the $$p$$-value depends on the rejection region of the test statistic. Indeed, given a sample $$X_1,\ldots,X_n$$, a test statistic $$T(X_1,\ldots,X_n)$$ and $$R_\alpha$$, a rejection region of size $$\alpha$$, then

$$p\text{-value} = \inf\{\alpha: T(X_1,\ldots,X_n)\in R_\alpha\}.$$

Thus the $$p$$-value can be interpreted as the smallest size at which we can reject $$H_0$$. Thus the $$p$$-value tells us how surprising is the value of the statistic when $$H_0$$ is true, to be interpreted as:

the lower the $$p$$-value, the more surprising is observed such a value under the model with $$H_0$$ being true.

Indeed, some authors refer to the $$p$$-value by the name observed significance level.

Now about the computation of the $$p$$-value, if $$t_n$$ is the observed test statistics then:

• if the rejection region is of the form $$\{X_1,\ldots,X_n: T(X_1,\ldots,X_n)\geq c\},$$ then the $$p$$-value is defined by $$\sup_{\theta\in\Theta_0} P_\theta(T(X_1,\ldots,X_n)\geq t_n);$$

• if the rejection region is of the form $$\{X_1,\ldots,X_n: T(X_1,\ldots,X_n)\leq c\},$$ then the $$p$$-value is defined by $$\sup_{\theta\in\Theta_0} P_\theta(T(X_1,\ldots,X_n)\leq t_n);$$

• if the rejection region is of the form $$\{X_1,\ldots,X_n: T(X_1,\ldots,X_n)\geq c_1\}\cup \{X_1,\ldots,X_n: T(X_1,\ldots,X_n)\geq c_2\},$$ then the $$p$$-value is defined by $$2\min\left(\sup_{\theta\in\Theta_0} P_\theta(T(X_1,\ldots,X_n)\leq t_n,\sup_{\theta\in\Theta_0} P_\theta(T(X_1,\ldots,X_n)\geq t_n\right).$$

• That is intuitive especially the usage of the word surprising. +1. Nov 11, 2022 at 14:05
• I think I used to interpret $p$-value in this fashion, and I think it reflects Fisher's thinking. However, the currently popular interpretation seems to be slightly different. I have tried to tackle the difference in these posts: $p$-value: Fisherian vs. contemporary frequentist definitions, Does $p$-value ever depend on the alternative? and Defining extremeness of test statistic and defining $p$-value for a two-sided test. Nov 11, 2022 at 17:28
• @Paradox: I think my explanation "Thus the $p$-value can be interpreted as the smallest size at which we can reject $𝐻_0$" along with the Nick Cox criticism on the choice of epsilon does. If you are NOT convinced, what about writing your own answer to it? Nov 11, 2022 at 20:09
• If we treat the null hypothesis in isolation and consider how surprising the test statistic is, one should be the most surprised to see an observation from the lowest density region. To realize that, it may help to think about a generic random variable with some weird null distribution, ideally one where the lowest density is not in the tails. (Thus not a $t$-test; there is one coincidence too many going on there.) When we add the alternative, it boils down to the ratio of the densities. Meanwhile, the values of the test statistic per se have nothing to say without the two competing densities. Nov 11, 2022 at 20:53
• @utobi, regarding the $t$-test, it is an unfortunate pedagogical example as the density of the test statistic declines symmetrically around zero. Thus we cannot distinguish between low density and high $|t_{obs}|$. But if we take the interpretation of the $t$ test statistic and try to apply it to another test where this is not the case, we are stuck. We thus learn it is the density, not the size of the statistic that defines extremeness – whether directly (Fisher's original interpretation) or in relation to the density under the alternative hypothesis (currently widespread interpretation). Nov 12, 2022 at 7:57

How is $$p$$-value actually defined?

Definition $$1.$$ (cf. $$\rm[I]$$) A $$p$$-value is a test statistic $$p:\mathcal X\to [0,1]$$ such that $$\mathbb P_\theta (p(\mathbf X) \leq \alpha)\leq\alpha,~~\forall\theta\in\Omega_\mathcal H, ~\forall\alpha\in(0,1).\tag{1.a}\label 1$$

Consider a series of nested tests $$\langle \varphi_\alpha\rangle$$ in the sense that $$\varphi_\alpha(x) \leq\varphi_{\alpha^\prime}(x)$$ for $$\alpha<\alpha^\prime.$$

Define $$\hat p:=\inf\{\alpha:\varphi_\alpha=1\}.\tag{1.b}\label b$$

Observation $$1.1.$$ $$\hat p$$ is a valid $$p$$-value.

Formally, if for a set of nested test functions $$\sup_{\theta\in\Omega_\mathcal H}\mathbb P_\theta(\varphi_\alpha(\mathbf X) \leq \alpha) \leq \alpha ~~\forall\alpha\in(0, 1), \tag 2\label 2$$

then for all $$u\in(0,1),$$ $$\mathbb P_\theta\left(\hat p\leq u\right) \leq u. \tag 3\label 3$$

$$\eqref 3$$ is easy to see (cf. $$\rm [II]$$ ) for $$\left\{\hat p\leq u\right\}$$ means $$\{\varphi_v(\mathbf X) =1\}$$ for all $$u Then, let $$v\to u.$$

$$\blacksquare$$

Now consider a test statistic $$W(\mathbf X)$$ whose large values indicate the rejection of $$\mathcal H.$$

Observation $$1.2.$$ (cf.$$\rm [I]$$) Define

$$p(\mathbf x) := \sup_{\theta\in\Omega_\mathcal H} \mathbb P_\theta(W(\mathbf X) \geq W(\mathbf x)).\tag{1.c}\label c$$ $$p(\mathbf x)$$ is also a valid $$p$$-value.

Notice that

\begin{align}p_\theta(\mathbf x) &= \mathbb P_\theta (W(\mathbf X) \geq W(\mathbf x))\\&= \mathrm F_\theta(-W(\mathbf x)),\tag 4\end{align}

which implies $$p_\theta(\mathbf x)$$ is stochastically greater than or equal to $$\mathcal U(0, 1).$$ Then as $$p(\mathbf x) \geq p_\theta(\mathbf x),$$ $$\eqref 1$$ follows.

$$\blacksquare$$

When one talks about $$p$$-value, they are basically meaning $$\eqref 1$$ or $$\eqref c$$ which as outlined above are genuine $$p$$-values.

The phrase as extreme as is essential to define $$p$$-value. Gloss over the definition and its equivalence as outlined above.

However, what does it imply intuitively? What does lower $$p$$-value mean? Why is the phrase necessary?

Over the years, there have been many CV posts dealing with the specifics. Please have a look at some of those:

$$\bullet$$ Why is smaller the p-value, larger is the significance?

$$\bullet$$ Does p-value ever depend on the alternative? (courtesy Richard Hardy)

$$\bullet$$ What is the meaning of p values and t values in statistical tests?

## References:

$$\rm [I]$$ Statistical Inference, George Casella, Roger L. Berger, Wadsworth, $$2002,$$ sec. $$8.3,$$ pp. $$397-398.$$

$$\rm [II]$$ Testing Statistical Hypotheses, E. L. Lehmann, Joseph P. Romano, Springer Science$$+$$Business Media, $$2005,$$ sec. $$3.3,$$ pp. $$63-64.$$

• Regarding as extreme as the one per $\mathcal{H}$, this might not be the usual formulation of extremeness. Here are a few related threads on the topic: Does $p$-value ever depend on the alternative?, Defining extremeness of test statistic and defining $p$-value for a two-sided test and $p$-value: Fisherian vs. contemporary frequentist definitions. Nov 11, 2022 at 17:16
• I must acknowledge @RichardHardy that that was hand wavy at best and to an extent superficial. However, what matters are the definitions you are working with. If there is no vacillation re that, you are free from ambiguities. Nevertheless, thanks for these links! Nov 11, 2022 at 17:26
• In fact, @RichardHardy, Ben's answer aptly describes the phrase in the form of a total ordering aka the evidentiary ordering which in our formalism is nothing but $\rm (1.c).$ That's beautiful indeed. Nov 11, 2022 at 18:28

Regarding the first question, you have got some good answers already. (You may want to check out the following threads, too, as I think they align with your thinking quite well: $$p$$-value: Fisherian vs. contemporary frequentist definitions, Does $$p$$-value ever depend on the alternative? and Defining extremeness of test statistic and defining $$p$$-value for a two-sided test).

Regarding the second question, you start with a Fisherian-esque interpretation by only looking at the null hypothesis and ignoring the alternative. This is pretty intuitive to me, but it does not seem to be fashionable anymore. However, you differ by suggesting to only look at the density at or around the test statistic – not the integral of all densities that are at most as high. The latter would be the Fisherian $$p$$-value*. It has the advantage of using a unified (and I think reasonably intuitive) scale between 0 and 1 which is based on ranking all possible test statistics from the most extreme (in the Fisherian sense, i.e. having the lowest likelihood*) to the least extreme (having the highest likelihood). Meanwhile, your approach would use a different scale for different tests (Student-$$t$$, $$\chi^2$$, $$F$$, ...), so we would have to develop intuition for each of them. While we can now quite easily judge a value between 0 and 1, there we would be dealing with all kinds of values on quite different scales – not quite as easy.

*Not everyone may agree on that; I had some discussion about it somewhere else on this site. Too bad Fisher is not here with us anymore to elaborate on his position.

Some further references:

• Completed chapter $14$ of Spanos: it was light and yet insightful read. Prior to that, I was never aware of the specifics of the schism between Fisher and NP formalism. Here is the crux: Fisher formulated a statement about the underlying statistical model aka the null. To lead credence to $H_0,$ he devised a distance function using pivotal function. Then to evaluate the worst case scenario for the null, he measured $p$-value. As he interpreted, small values would indicate either the observed realisation was a rare event or the null is invalid, the former being practically impossible. Nov 13, 2022 at 12:47
• Fisher is inferential in that one is focused on to what extent the sample realisation leads credence to $H_0.$ NP shifts the sole focus from $H_0$ and provides a logical basis of choosing the test statistic based on which, one has to choose between two rival hypotheses. Critical difference is not the absence of an alternative in Fisher's but rather how it was interpreted: Fisher's $H_{1F}: f(\mathbf x) \in [\mathcal P\setminus\Phi_0]$ which is broader than NP's in that it scours without the boundary of the postulated model. Significance level & $p$-value are two different things and play Nov 13, 2022 at 12:47
• @User1865345, Spanos has written extensively on issues like that (history, philosophy, foundations and their interpretations). I have enjoyed many of his paper and learned a lot. He blogs, too: errorstatistics.com/tag/aris-spanos. Nov 13, 2022 at 13:09
• Well. He indeed knows how to present the historical context in a comprehensive manner without losing the soul. Reading sources like his is bit of a fresh air from those monotonous conventional literatures. And great! He has a blog too! Nov 13, 2022 at 13:13
• He has also been referenced on Cross Validated (a few times by me, but also by others): stats.stackexchange.com/search?q=spanos. Nov 13, 2022 at 13:18

Likelihood is a pain to work with.

For starters, likelihood can only go down as you collect more data. Indeed, if your alternate hypothesis is "My coin is biased in favour of tails" and you flip it 0 times, the likelihood of getting zero heads is 100%! Actually doing the experiment can only ever hurt your likelihood value from there.

Second, likelihood is non-exclusive. The likelihood of zero heads from zero throws is also 100% under the null, as well as under the hypothesis "It is a magic coin that always lands how I want." Even if we actually flip our coin and get tails, we're scoring a 75% likelihood. But the null hypothesis still scores a 50% likelihood. It isn't a straightforward battle between null and alternate.

Lastly, likelihood calculations are incredibly sensitive to the formulation of the hypothesis. In practice you often don't know ahead of time (and stating your hypothesis ahead of time is important!) what exact numbers to put in. If you suspect someone is cheating with a coin biased for tails, it's a pain to have to say whether it's a 70% tails or a 75% tails coin. Now suppose you take that coin and flip it 1000 times, getting exactly 748 tails. Under the 75% hypothesis your likelihood of that result is an astounding 2.9%, but under a 70% hypothesis it would be 0.01%. Under the hypothesis we really want, "It's biased to somewhere in the 65-80 zone" it's just messy to define and calculate.

The intuitive measure you're after is "Probability of our hypothesis." Unfortunately, it generally is not a value we can actually calculate from our experiment. It depends too much on things like priors (Fair coins are more common than magic coins, but how much?), and the huge space of possibility of other hypotheses (e.g. what if the coin is fair but the flipper is trained?).

So, we make do with p values as our first filter. To be clear, a tiny p value does not mean that the alternate hypothesis is true. All it means is that the researcher has done enough work to beat the trivial "It's a fluke" standard that they've earned the right to have their work examined.