# p-value with multimodal PDF of a test statistic

I have opened a thread about p-value under the title "Understanding p-value" and gotten two answers and some comments. I think my questions in the thread is somewhat diverse and want to clarify my question more explicitly based on the discussion in the thread. Two different definitions of the p-value were suggested in the thread.

definition 1

The p-value is $\int_{\{x\,:\,f(x) \le f(x_o)\}} f$.

definition 2

The p-value is $\int_{\{x\,:\,x_o \le x\}} f$.

In both of the definition, $f$ is the PDF of a chosen test statistic under the null hypothesis and $x_o$ is the observed value of the test statistic. I think the two definitions are clear and complete enough. (The p-value concerns data, a null hypothesis and a chosen statistic only. It does not concern the alternative hypothesis or other things.)

The role of the p-value is to quantify how likely the observation is under the null hypothesis. Small p-value means the observed data is weird (ie. unlikely) under the null hypothesis and the assumed null hypothesis should be rejected.

The definition 1 measures this weirdness in terms of $f(x_o)$, the probability density of the observed test statistic. So the definition integrates $f$ over the values of the test statistic that have smaller probability density (ie. more weird) than the observed one.

The definition 2 measures the weirdness in terms of the distance of $x_o$ from the most likely value of the test statistic, if the most likely value is well defined. So the definition integrates $f$ over the values from the observed one to tail (ie. more weird region).

If $f$ is unimodal, both of the two definitions seem reasonable. If $f$ is multimodal, however, I think the definition 2 is not reasonable. For an example, let's assume that $f$ is bimodal and $x_o$ is somewhere in the low probability density region between the two peaks. Then the most likely value is not well defined and the distance of $x_o$ from the most likely value cannot be reasonable measure of the weirdness. The p-value calculated along the definition 2 may be very large, whereas the observation $x_o$ is obviously weird because of its low probability density. The definition 1 still works in this case as it gives small p-value.

I am not a statistician and I don't know which one of the definitions is "the right one" that statisticians usually use. Most of the materials I have seen before explain p-value in the sense of the definition 2. But, I encountered the definition 1 in Zag's answer of the old thread for the first time and was persuaded. What is the exact definition of the p-value? If it is not the definition 1, I'd like to know rationale for the right one and shortcomings of the definition 1.

-
By the way I think you mean $\int_{\{x\,:\,f(x) \leq f(x_o)\}} f$ in Definition 1. The value of $x_o$ might be 10, in which case $f(x)$ would always be less. – Scortchi Dec 3 '12 at 18:42
@Scortchi, you are right, I edited the definition. – JDL Dec 4 '12 at 0:26
what makes a test statistic 'more extreme' is the ordering imposed on the sample space by the values of the statistic - in particular, that the values most consistent with the alternative are the 'most extreme'. The only test I am aware of that orders by $f$ is Fisher's 'exact' test on 2x2 tables, but IIRC that has a one-to-one relationship with a natural ordering of a statistic – Glen_b Dec 4 '12 at 2:59

I think all this is way too much "p-value centered".

You have to remember what tests are really about: rejecting a null hypothesis with a given value for the α risk. The $p$-value is just a tool for this. In the most general situation, you have build a statistic $T$ with known distribution under the null hypothesis ; and to chose a rejection region $A$ so that $\mathbb P_0(T \in A) = \alpha$ (or at least $\le \alpha$ is equality is impossible). P-values are just a convenient way to chose $A$ in many situations, saving you the burden of making a choice. It's an easy recipe, that’s why is so popular, but you shouldn’t forget about what’s going on.

As $p$-values are computed from $T$ (with something like $p = F(T)$ they are also statistics, with uniform $\mathcal U(0,1)$ distribution under the null. If they behave well, they tend to have low values under the alternative, and you reject the null when $p \le\alpha$. The rejection region $A$ is then $A = F^{-1}( (0,\alpha) )$.

OK, I waved my hands long enough, it’s time for examples.

A classical situation with a unimodal statistic

Assume that you observe $x$ drawn from $\mathcal N(\mu,1)$, and want to test $\mu = 0$ (two-sided test). The usual solution is to take $t = x^2$. You know $T \sim \chi^2(1)$ under the null, and the p-value is $p = \mathbb P_0( T \ge t)$. This generates the classical symmetrical rejection region shown below for $\alpha = 0.1$.

In most situations, using the $p$-value leads to the "good" choice for the rejection region.

A fancy situation with a bimodal statistic

Assume that $\mu$ is drawn from an unknown distribution, and $x$ is drawn from $\mathcal N(\mu,1)$. Your null hypothesis is that $\mu = -4$ with probability $1\over 2$, and $\mu = 4$ with probability $1\over 2$. Then you have a bimodal distribution of $X$ as displayed below. Now you can't rely on the recipe: if $x$ is close to 0, let’s say $x = 0.001$... you sure want to reject the null hypothesis.

So we have to make a choice here. A simple choice will be to take a rejection region of the shape $$A = (-\infty, -4-a) \cup (-4+a, 4-a) \cup (4+a, \infty)$$ width $0< a$, as displayed below (with the convention that if $a \ge 4$, the central interval is empty). The natural choice is in fact to take a rejection region of the form $A = \{ x \>:\> f(x) < c \}$ where $f$ is the density of $X$, but here it is almost the same.

After a few computations, we have $\newcommand{\erf}{F}$ $$\mathbb P( X \in A ) = \erf(-a)+\erf(-8-a) + \mathbf 1_{\{a<4\}} \left( \erf(8-a)-\erf(a)\right)$$ where $F$ is the cdf of a standard gaussian variable. This allows to find an appropriate threshold $a$ for any value of $\alpha$. Now to retrieve a $p$-value that give an equivalent test, from an observation $x$, one take $a = \min( |4-x|, |-4-x| )$, so that $x$ is at the border of the corresponding rejection region ; and $p = \mathbb P( X \in A )$, with the above formula.

Post-Scriptum If you let $T = \min( |4-X|, |-4-X| )$, you transform $X$ into a unimodal statistic, and you can take the $p$-value as usual.

-
I agree that the overall volume of conversation on $p$-values is disproportionate to the usefulness of the tool when it comes to estimating statistical models. Regarding the question discussed here, the basic assumption of the test is that the random variable will be normally distributed as $N$ increases, in which case the distribution will be unimodal and the bimodality is only a sample anomaly. The explanation above is good on what to do under that assumption. If that assumption is violated, then neither definition should be considered relevant. – Fr. Dec 8 '12 at 15:15

Actually both of you definitions work in different cases, it depends on how you define your null hypothesis (which is often affected by the way you state your alternative hypothesis, so it does matter).

If your null hypothesis is strictly that the parameter(s) equal a given value (or set of values, 1 per parameter), e.g. $H_0: \mu=\mu_0$ then your first definition works (well with $f(x) \le f(x_0)$). This is the 2-tailed test in the traditional simple statistics cases.

But often we are interested only in the alternative being in a certain direction, the one-tailed test case. E.g. If I want to prove that my new pain reliever is better than aspirin (takes less time for the headache to go away on average) then I am only interested in 1 tail and my alternative would be $H_a: \mu < \mu_0$ (if I prove that my new medicine takes longer then it will not help my advertising). This leads to the null hypothesis being $H_0: \mu \ge \mu_0$ even though we often write it as $H_0: \mu = \mu_0$. In this case we only want to look at the possible $x$ values in a certain region, so more like definition 2.

In practice, most common test statistics follow a unimodal distribution (or are close enough) under the null hypothesis, so both definitions are the same. The only common case I know of where all possible cases with lower likelihood are included in the p-value is Fisher's exact test for tables biger than $2\times2$.

So to sum up. Your thinking is generally correct, cases that you suggest are just rare enough that most books/classes only present the simpler version.

-
But there's a bit more to it than that. The example I used for the original question was the likelihood ratio test for goodness of fit in the binomial case, with a binomial probability of 0.85 under the null & a sample size of 10. It's a point null, yet the pdf of the test statistic is multimodal. – Scortchi Dec 3 '12 at 21:22
I actually doubt the use of 1-sided test like your example although many texts suggest such test. Your test statistic do have probabilities for $\mu < \mu_0$, even though you assume $\mu \ge \mu_0$. How do you deal this discrepancy? I think simply ignoring some part of the distribution is not a right way. IMHO, you have to accept the distribution as it is and use 2-sided test (or using definition 1 is better), this approach submits handicap from the discrepancy and at least more strict than 1-sided one (It always gives larger p-value). – JDL Dec 4 '12 at 1:21
@JDL, If were to claim that I am taller than everyone in your family and we wanted to test that claim then we could either compare my height to the height of each member of your family, or simpler would be to just find the tallest person in your family then compare my height to theirs. It is similar with the 1 sided test, we use the equality for the null distribution and if we reject that value (showing that we believe the true value to be less) then we have also rejected every value greater than that null value. – Greg Snow Dec 5 '12 at 20:12
@Scortchi, I don't see the example that you refer to, can you give more detail? – Greg Snow Dec 5 '12 at 20:30
@Scortchi, thanks for the link. I believe that we agree. Part of my point above is that for most common tests taught in intro classes (which does not include LR in my experiance) the distribution used (possibly an approximation) is unimodal and so the 2 ways of determining the p-value (with the tail issue specified) are equivalent, so most intro classes don't bother to talk about the more complicated (but more correct in general) definition. You give an example where it is not unimodal and more thought and better understanding are required. – Greg Snow Dec 6 '12 at 16:58

This is really two questions:

(1) What is the definition of a p-value?

Answer: Definition 2—the probability under the null hypothesis of getting a value of the test statistic greater than or equal to that observed. (As @whuber pointed out, it needs some qualification: in the case of a composite null hypothesis the probability involved is the maximum probability over every point null in that set; the probability of what's sometimes called the proximal null hypothesis.)

(2) Should a test statistic strictly increase with decreasing probability under the null hypothesis ?

I have tried to answer this in responses to your previous post. (Answer: not always.) Hope someone can explain more clearly if needed. At least note here that many commonly used test statistics don't. You have ...

(a) test statistics ordered by the probability under the null: Fisher's Exact Test, as Greg Snow notes, & the test for a binomial parameter given by Zag.

(b) test statistics ordered by likelihood ratio (sometimes but not always giving the same ordering as (a)): my binomial goodness-of-fit test example.

(c) test statistics chosen for maximum power against specified alternatives (sometimes but not always giving the same ordering as (a) and/or (b), as I think RobertF was getting at): 'The Emperor's new tests', Perlman & Wu (1999), together with the comments & rejoinder, is very interesting (though difficult).

If you read the paper by Christensen that Zag linked to, you will see that in the first example he writes "With only this information, one must use the density itself to determine which data values seem weird and which do not". The clear implication is that with more information you needn't necessarily use the density itself to determine which data values seem weird and which do not.

In response to @whuber's comment ...

The likelihood ratio test is in fact a good example of Defn 2's being used. The p-value in this case is just the probability (under the null) of the the likelihood ratio's being larger or equal to that observed.

As an elementary example, you can test two hypotheses for the probability of success in a Bernoulli trial : $$H_0: \theta = 0.55$$ $$H_1: \theta = 0.35$$

Nine independent trials give $t$ successes:

$$\newcommand{\pr}{\mathrm{Pr}}\begin{array}{cccc} t & \pr(t|H_0) & \pr(t|H_1) & \frac{\pr(t|H_1)}{\pr(t|H_0)}=x\\ 0 & 0.00076 & 0.02071 & 27.372\\ 1 & 0.00832 & 0.10037 & 12.060\\ 2 & 0.04069 & 0.21619 & 5.3128\\ 3 & 0.11605 & 0.27162 & 2.3406\\ 4 & 0.21276 & 0.21939 & 1.0312\\ 5 & 0.26004 & 0.11813 & 0.4543\\ 6 & 0.21188 & 0.04241 & 0.2001\\ 7 & 0.11099 & 0.00979 & 0.0882\\ 8 & 0.03391 & 0.00132 & 0.0389\\ 9 & 0.00461 & 0.00008 & 0.0171 \end{array}$$

The likelihood ratio $x$ is your test statistic.

Using Defn 1 to get a p-value, you have to add up all the probabilities (under the null) for less (or equally) probabable values of $x$ than that observed. So, observing $t = 2$, you'd add up those for $2$, $8$, $1$, $9$, & $0$ successes to give $0.04069 + 0.03391 + 0.00832 + 0.00461 + 0.00076 = 0.08829$

Using Defn 2, you add up all the probabilities (under the null) for values of $x$ larger than (or equal to) that observed. So, observing $t = 2$, $x$ is larger for $0$ & $1$ successes so you add their probability under the null to that of $2$ successes to give a p-value of $0.04069 + 0.00832 + 0.00076 = 0.04977$.

It's clear that the latter procedure is the likelihood ratio test as usually understood, & that defined by the Neyman–Pearson lemma.

-
Here is another quote from the paper : if you have another statistic you want to base a test on, the Fisherian tester is happy to oblige. To quote Fisher (1956, p. 49), the hypothesis should be rejected "if any relevant feature of the observational record can be shown to [be] sufficiently rare.” After all, if the null model is correct, it should be able to withstand any challenge. I agree to the sentences. Whatever statistic is used, its probability density is at our hand and I think the density must be considered in the way of definition 1. – JDL Dec 4 '12 at 0:57
@whuber, But Defn 1 doesn't use relative likelihood, only probability under the null. – Scortchi Dec 17 '12 at 15:35
Excellent point! I misread the formula as comparing the null with an alternative probability. (Now your previous comments make better sense to me :-).) In that case, I have to maintain that neither definition in the question is correct. A p-value makes sense only in the context of a clearly circumscribed alternative hypothesis, whence no definition based solely on the null distribution can accurately express the concept. – whuber Dec 17 '12 at 15:48
@whuber, Defn 2 accounts for the distinction between one- & two-tailed tests against a point null hypothesis: e.g. for testing a normal mean; if $z = \frac{\bar{x_i}-\mu_0}{\sigma}$, the test statistic is $z$, $-z$, or $|z|$ as appropriate. (Yes, the choice of test statistic relies on how the alternative hypothesis is formulated.) But you're right that it needs expanding to cover composite nulls. – Scortchi Dec 17 '12 at 17:11
@whuber, And in regard to your last sentence - I'm unaware of any justification of, say, the Wilcoxon rank sum test that makes any appeal to a likelihood ratio. Except insofar as the choice of test statistic is intended to measure discrepancy of data with the null in the direction of a vaguely conceived alternative. – Scortchi Dec 17 '12 at 17:19