It is well established that p-values are uniformly distributed when the null hypothesis is true. This follows from the definition of a p-value
The probability of observing a value (or more extreme one) when those values are drawn from the known, and fixed distribution (i.e. null is true).
This fact allows for a range of follow up analysis when looking at distribution of pvalues.
http://varianceexplained.org/statistics/interpreting-pvalue-histogram/ examples of looking at p-value histograms.
However, I am concerned with a different probability. Instead of the proportion of observations that are 'more extreme'. I would like to know the proportion of observations that are 'more rare'.
It is true that 'more extreme' implies 'more rare' however, 'more rare' does not imply 'more extreme' -- particularly for multimodal distributions under the null as shown in the 2 images below. An observation could be near the mean and still be a rare observation from a low density portion of the null distribution.
One sided p-value $$P(X > x | H)$$
For my 'd-values': $$P(\theta(X) \le \theta(x) | H)$$
For a density function theta (which in my case comes from a simple univarate KDE)
Questions:
1) What are these "d-values" called? I can't be the first person to have this question?
2) How are these "d-values" distributed under Ho?
Let $0 \le \beta \le \max_x(\theta(x))$ (the density of the highest mode)
$P(\theta(x) \le 0) = 0$
$P(\theta(x) \le \max_x(\theta(x))) = 1$
$P(\theta(x) \le \beta) = {}$??
This is kind of like a vertical integration over density values, but leaving out any density > threshold.
3) Does the distribution of 2 hold no matter what form the distribution of observations is under Ho? (It does for p-values -> uniform).