# Distribution of "p-value-like" quantities under null hypothesis

It is well established that p-values are uniformly distributed when the null hypothesis is true. This follows from the definition of a p-value

The probability of observing a value (or more extreme one) when those values are drawn from the known, and fixed distribution (i.e. null is true).

This fact allows for a range of follow up analysis when looking at distribution of pvalues.

However, I am concerned with a different probability. Instead of the proportion of observations that are 'more extreme'. I would like to know the proportion of observations that are 'more rare'.

It is true that 'more extreme' implies 'more rare' however, 'more rare' does not imply 'more extreme' -- particularly for multimodal distributions under the null as shown in the 2 images below. An observation could be near the mean and still be a rare observation from a low density portion of the null distribution.

One sided p-value $$P(X > x | H)$$

For my 'd-values': $$P(\theta(X) \le \theta(x) | H)$$

For a density function theta (which in my case comes from a simple univarate KDE)

Questions:

• 1) What are these "d-values" called? I can't be the first person to have this question?

Let $0 \le \beta \le \max_x(\theta(x))$ (the density of the highest mode)

$P(\theta(x) \le 0) = 0$

$P(\theta(x) \le \max_x(\theta(x))) = 1$

$P(\theta(x) \le \beta) = {}$??

This is kind of like a vertical integration over density values, but leaving out any density > threshold.

• 3) Does the distribution of 2 hold no matter what form the distribution of observations is under Ho? (It does for p-values -> uniform).

• Inferring rarity from the value of the density function for $X$ is perplexing, because then rarity will not be invariant to even monotonic transformations of $X$. Commented Apr 13, 2018 at 8:53
• Perhaps 'rare' is the wrong term. Happy to adjust the question with a more appropriate one. Commented Apr 13, 2018 at 9:21
• No need; that was just an observation. Commented Apr 13, 2018 at 10:02
• Commented Apr 13, 2018 at 10:12
• I believe this is related to confidence intervals... when one wishes to find the smallest range/set with a specific condifence. Commented Apr 17, 2018 at 13:28

Let $f$ be the density of $X$. You are concerned about the distribution of ''d-values'' $$d = P( f(X) < f(x_{obs}))$$ when $x_{obs}$ is drawn in the distribution of $X$.

Let's construct an other random variable by transforming $X$ : $Y = f(X)$, and let $y_{obs} = f(x_{obs})$. Then in fact you're looking at the distribution of $$P(Y < y_{obs})$$ when $y_{obs}$ is drawn in the distribution of $Y$.

It is then the uniform distribution.

A quick numerical experiment

Consider a mixture of two Gaussian with variance 1 and means 0 and 4. Its density looks like

Now for the numerical experiment:

# a reference sample to compute d values
X_ref <- c( rnorm(1e4), rnorm(1e4, mean = 4) )

# a set of observations
x_obs <- c( rnorm(1e4), rnorm(1e4, mean = 4) )

# the d-values
d <- sapply(x_obs, function(x) mean(f(x) < f(X_ref)) )

plot(ppoints(2e4), sort(d), pch = ".")


• There's no problem in this case, it is like taking bilateral p-values. I am a little more worried by bimodal distributions, but a quick numerical experiment show it's ok. Commented Apr 13, 2018 at 7:39
• What about X ~ uniform(0.0,2) (rectangle with height 1/2) The d/p-value jumps from 0 to 1 when the density crosses 0.5. Maybe this is equivalent to a point-mass for traditional p-value calcs. Commented Apr 13, 2018 at 7:42
• In that case $f(Y)$ is a constant random variable. You can update my ''It is then the uniform distribution" with ''provided that the distribution of $Y$ is continuous with respect to the Lebesgue measure''. Commented Apr 13, 2018 at 7:48

Short answer: The statistic you are referring to may just be the p-value (depending on the ordering of evidence for the null vs alternative. Don't assume that p-values are for the area that is "extreme" in the sense of having the highest magnitude values (i.e., the tail area).

Longer answer: Every hypothesis test involves an implicit ordering of possible outcomes on an ordinal scale from those that are more conducive to the null hypothesis to those that are more conducive to the alternative hypothesis. This implicit ordering is captured in the test-statistic and its order as a measure of evidence. The p-value is defined as the probability of observing evidence at least as conducive to the alternative hypothesis as what was actually observed, assuming the null hypothesis is actually true.

It is certainly not the case that observations that are "more extreme" in the sense of being high magnitude (i.e., in the tail of a distribution) are necessarily more conducive to the alternative hypothesis than values like the one you highlight above. In fact, the most common measure used for the test statistic is the likelihood-ratio statistic comparing the null and alternative hypotheses. This test statistic orders the possible outcomes so that outcomes with lower relative likelihood for the null hypothesis (compared to the alternative) constitute greater evidence against the null. In this test, the p-value is the values that are "most extreme" in favour of the alternative, which is the set of values with low likelihood ratio:

$$p(x) \equiv \mathbb{P}( LR(X) \leqslant LR(x) | H_0 ) \quad \quad LR(x) = \frac{L_0(x)}{L_A(x)},$$

where $L_0 \equiv \sup_{\theta \in \Theta_0} L_x(\theta)$ and $L_A \equiv \sup_{\theta \in \Theta_A} L_x(\theta)$ are the likelihoods under the null and alternative models. Now, if you were to show the likelihood ratio in your function instead of the null distribution, then the area you are calling a "d-value" would actually just be the p-value of the test. The critical region for the test would be a disjointed set of intervals on the horizontal axis.

The value you have highlighted in your diagram is based solely on looking at the null distribution, which means we cant really see what the proper ordering of evidence in the test is. But you shouldn't assume that the ordering is such that higher magnitude values are more evidence for the alternative (particularly with a multi-modal null distribution. Once you specify the likelihood for the alternative, you will be able to determine the ordering of evidence implicit in use of the likelihood ratio statistic and this will tell you what constitutes a "more extreme" value in your test.

• I believe my alternative is 'p-values are not uniform'. This can be measured by comparing the number of pvalues less than alpha and N*alpha for alpha between 0 and 1. (Higher Criticism). It can also be measured by the KL divergence between the observed-pvalues and the uniform distribution (Berk-Jones). Section 2.3 of cs.cmu.edu/~neill/papers/mcfowland13a.pdf but mostly the first half of page 1540. Commented Apr 13, 2018 at 7:58
• (+1) @Eruditio: Your alternative hypothesis, if you have one in mind, will be a distribution or family of distributions governing the data-generating-process other than that stipulated by the null. Commented Apr 13, 2018 at 8:47