Distribution of "p-value-like" quantities under null hypothesis

Question

It is well established that p-values are uniformly distributed when the null hypothesis is true. This follows from the definition of a p-value

The probability of observing a value (or more extreme one) when those values are drawn from the known, and fixed distribution (i.e. null is true).

This fact allows for a range of follow up analysis when looking at distribution of pvalues.

http://varianceexplained.org/statistics/interpreting-pvalue-histogram/ examples of looking at p-value histograms.

However, I am concerned with a different probability. Instead of the proportion of observations that are 'more extreme'. I would like to know the proportion of observations that are 'more rare'.

It is true that 'more extreme' implies 'more rare' however, 'more rare' does not imply 'more extreme' -- particularly for multimodal distributions under the null as shown in the 2 images below. An observation could be near the mean and still be a rare observation from a low density portion of the null distribution.

One sided p-value $$P(X > x | H)$$

For my 'd-values': $$P(\theta(X) \le \theta(x) | H)$$

For a density function theta (which in my case comes from a simple univarate KDE)

Questions:

• 1) What are these "d-values" called? I can't be the first person to have this question?

• 2) How are these "d-values" distributed under Ho?

  Let $0 \le \beta \le \max_x(\theta(x))$ (the density of the highest mode)

  $P(\theta(x) \le 0) = 0$

  $P(\theta(x) \le \max_x(\theta(x))) = 1$

  $P(\theta(x) \le \beta) = {}$??

  This is kind of like a vertical integration over density values, but leaving out any density > threshold.

• 3) Does the distribution of 2 hold no matter what form the distribution of observations is under Ho? (It does for p-values -> uniform).

Answer 1

Let $f$ be the density of $X$. You are concerned about the distribution of ''d-values'' $$d = P( f(X) < f(x_{obs}))$$ when $x_{obs}$ is drawn in the distribution of $X$.

Let's construct an other random variable by transforming $X$ : $Y = f(X)$, and let $y_{obs} = f(x_{obs})$. Then in fact you're looking at the distribution of $$P(Y < y_{obs})$$ when $y_{obs}$ is drawn in the distribution of $Y$.

It is then the uniform distribution.

A quick numerical experiment

Consider a mixture of two Gaussian with variance 1 and means 0 and 4. Its density looks like

Now for the numerical experiment:

# a reference sample to compute d values
X_ref <- c( rnorm(1e4), rnorm(1e4, mean = 4) )

# a set of observations
x_obs <- c( rnorm(1e4), rnorm(1e4, mean = 4) )

# the d-values
d <- sapply(x_obs, function(x) mean(f(x) < f(X_ref)) )

plot(ppoints(2e4), sort(d), pch = ".")

Answer 2

Short answer: The statistic you are referring to may just be the p-value (depending on the ordering of evidence for the null vs alternative. Don't assume that p-values are for the area that is "extreme" in the sense of having the highest magnitude values (i.e., the tail area).

Longer answer: Every hypothesis test involves an implicit ordering of possible outcomes on an ordinal scale from those that are more conducive to the null hypothesis to those that are more conducive to the alternative hypothesis. This implicit ordering is captured in the test-statistic and its order as a measure of evidence. The p-value is defined as the probability of observing evidence at least as conducive to the alternative hypothesis as what was actually observed, assuming the null hypothesis is actually true.

It is certainly not the case that observations that are "more extreme" in the sense of being high magnitude (i.e., in the tail of a distribution) are necessarily more conducive to the alternative hypothesis than values like the one you highlight above. In fact, the most common measure used for the test statistic is the likelihood-ratio statistic comparing the null and alternative hypotheses. This test statistic orders the possible outcomes so that outcomes with lower relative likelihood for the null hypothesis (compared to the alternative) constitute greater evidence against the null. In this test, the p-value is the values that are "most extreme" in favour of the alternative, which is the set of values with low likelihood ratio:

$$p(x) \equiv \mathbb{P}( LR(X) \leqslant LR(x) | H_0 ) \quad \quad LR(x) = \frac{L_0(x)}{L_A(x)},$$

where $L_0 \equiv \sup_{\theta \in \Theta_0} L_x(\theta) $ and $L_A \equiv \sup_{\theta \in \Theta_A} L_x(\theta) $ are the likelihoods under the null and alternative models. Now, if you were to show the likelihood ratio in your function instead of the null distribution, then the area you are calling a "d-value" would actually just be the p-value of the test. The critical region for the test would be a disjointed set of intervals on the horizontal axis.

The value you have highlighted in your diagram is based solely on looking at the null distribution, which means we cant really see what the proper ordering of evidence in the test is. But you shouldn't assume that the ordering is such that higher magnitude values are more evidence for the alternative (particularly with a multi-modal null distribution. Once you specify the likelihood for the alternative, you will be able to determine the ordering of evidence implicit in use of the likelihood ratio statistic and this will tell you what constitutes a "more extreme" value in your test.