p-value with multimodal PDF of a test statistic

Question

I have opened a thread about p-value under the title "Understanding p-value" and gotten two answers and some comments. I think my questions in the thread is somewhat diverse and want to clarify my question more explicitly based on the discussion in the thread. Two different definitions of the p-value were suggested in the thread.

definition 1

The p-value is $\int_{\{x\,:\,f(x) \le f(x_o)\}} f$.

definition 2

The p-value is $\int_{\{x\,:\,x_o \le x\}} f$.

In both of the definition, $f$ is the PDF of a chosen test statistic under the null hypothesis and $x_o$ is the observed value of the test statistic. I think the two definitions are clear and complete enough. (The p-value concerns data, a null hypothesis and a chosen statistic only. It does not concern the alternative hypothesis or other things.)

The role of the p-value is to quantify how likely the observation is under the null hypothesis. Small p-value means the observed data is weird (ie. unlikely) under the null hypothesis and the assumed null hypothesis should be rejected.

The definition 1 measures this weirdness in terms of $f(x_o)$, the probability density of the observed test statistic. So the definition integrates $f$ over the values of the test statistic that have smaller probability density (ie. more weird) than the observed one.

The definition 2 measures the weirdness in terms of the distance of $x_o$ from the most likely value of the test statistic, if the most likely value is well defined. So the definition integrates $f$ over the values from the observed one to tail (ie. more weird region).

If $f$ is unimodal, both of the two definitions seem reasonable. If $f$ is multimodal, however, I think the definition 2 is not reasonable. For an example, let's assume that $f$ is bimodal and $x_o$ is somewhere in the low probability density region between the two peaks. Then the most likely value is not well defined and the distance of $x_o$ from the most likely value cannot be reasonable measure of the weirdness. The p-value calculated along the definition 2 may be very large, whereas the observation $x_o$ is obviously weird because of its low probability density. The definition 1 still works in this case as it gives small p-value.

I am not a statistician and I don't know which one of the definitions is "the right one" that statisticians usually use. Most of the materials I have seen before explain p-value in the sense of the definition 2. But, I encountered the definition 1 in Zag's answer of the old thread for the first time and was persuaded. What is the exact definition of the p-value? If it is not the definition 1, I'd like to know rationale for the right one and shortcomings of the definition 1.

Answer 1

I think all this is way too much "p-value centered".

You have to remember what tests are really about: rejecting a null hypothesis with a given value for the α risk. The $p$-value is just a tool for this. In the most general situation, you have build a statistic $T$ with known distribution under the null hypothesis ; and to chose a rejection region $A$ so that $\mathbb P_0(T \in A) = \alpha$ (or at least $\le \alpha$ is equality is impossible). P-values are just a convenient way to chose $A$ in many situations, saving you the burden of making a choice. It's an easy recipe, that’s why is so popular, but you shouldn’t forget about what’s going on.

As $p$-values are computed from $T$ (with something like $p = F(T)$ they are also statistics, with uniform $\mathcal U(0,1)$ distribution under the null. If they behave well, they tend to have low values under the alternative, and you reject the null when $p \le\alpha$. The rejection region $A$ is then $A = F^{-1}( (0,\alpha) )$.

OK, I waved my hands long enough, it’s time for examples.

A classical situation with a unimodal statistic

Assume that you observe $x$ drawn from $\mathcal N(\mu,1)$, and want to test $\mu = 0$ (two-sided test). The usual solution is to take $t = x^2$. You know $T \sim \chi^2(1)$ under the null, and the p-value is $p = \mathbb P_0( T \ge t)$. This generates the classical symmetrical rejection region shown below for $\alpha = 0.1$.

In most situations, using the $p$-value leads to the "good" choice for the rejection region.

A fancy situation with a bimodal statistic

Assume that $\mu$ is drawn from an unknown distribution, and $x$ is drawn from $\mathcal N(\mu,1)$. Your null hypothesis is that $\mu = -4$ with probability $1\over 2$, and $\mu = 4$ with probability $1\over 2$. Then you have a bimodal distribution of $X$ as displayed below. Now you can't rely on the recipe: if $x$ is close to 0, let’s say $x = 0.001$... you sure want to reject the null hypothesis.

So we have to make a choice here. A simple choice will be to take a rejection region of the shape $$ A = (-\infty, -4-a) \cup (-4+a, 4-a) \cup (4+a, \infty) $$ width $0< a$, as displayed below (with the convention that if $a \ge 4$, the central interval is empty). The natural choice is in fact to take a rejection region of the form $A = \{ x \>:\> f(x) < c \}$ where $f$ is the density of $X$, but here it is almost the same.

After a few computations, we have $\newcommand{\erf}{F}$ $$\mathbb P( X \in A ) = \erf(-a)+\erf(-8-a) + \mathbf 1_{\{a<4\}} \left( \erf(8-a)-\erf(a)\right) $$ where $F$ is the cdf of a standard gaussian variable. This allows to find an appropriate threshold $a$ for any value of $\alpha$. Now to retrieve a $p$-value that give an equivalent test, from an observation $x$, one take $a = \min( |4-x|, |-4-x| )$, so that $x$ is at the border of the corresponding rejection region ; and $p = \mathbb P( X \in A )$, with the above formula.

Post-Scriptum If you let $T = \min( |4-X|, |-4-X| )$, you transform $X$ into a unimodal statistic, and you can take the $p$-value as usual.

Answer 2

Actually both of you definitions work in different cases, it depends on how you define your null hypothesis (which is often affected by the way you state your alternative hypothesis, so it does matter).

If your null hypothesis is strictly that the parameter(s) equal a given value (or set of values, 1 per parameter), e.g. $H_0: \mu=\mu_0$ then your first definition works (well with $f(x) \le f(x_0)$). This is the 2-tailed test in the traditional simple statistics cases.

But often we are interested only in the alternative being in a certain direction, the one-tailed test case. E.g. If I want to prove that my new pain reliever is better than aspirin (takes less time for the headache to go away on average) then I am only interested in 1 tail and my alternative would be $H_a: \mu < \mu_0$ (if I prove that my new medicine takes longer then it will not help my advertising). This leads to the null hypothesis being $H_0: \mu \ge \mu_0$ even though we often write it as $H_0: \mu = \mu_0$. In this case we only want to look at the possible $x$ values in a certain region, so more like definition 2.

In practice, most common test statistics follow a unimodal distribution (or are close enough) under the null hypothesis, so both definitions are the same. The only common case I know of where all possible cases with lower likelihood are included in the p-value is Fisher's exact test for tables biger than $2\times2$.

So to sum up. Your thinking is generally correct, cases that you suggest are just rare enough that most books/classes only present the simpler version.

Answer 3

This is really two questions:

(1) What is the definition of a p-value?

Answer: Definition 2—the probability under the null hypothesis of getting a value of the test statistic greater than or equal to that observed. (As @whuber pointed out, it needs some qualification: in the case of a composite null hypothesis the probability involved is the maximum probability over every point null in that set; the probability of what's sometimes called the proximal null hypothesis.)

(2) Should a test statistic strictly increase with decreasing probability under the null hypothesis ?

I have tried to answer this in responses to your previous post. (Answer: not always.) Hope someone can explain more clearly if needed. At least note here that many commonly used test statistics don't. You have ...

(a) test statistics ordered by the probability under the null: Fisher's Exact Test, as Greg Snow notes, & the test for a binomial parameter given by Zag.

(b) test statistics ordered by likelihood ratio (sometimes but not always giving the same ordering as (a)): my binomial goodness-of-fit test example.

(c) test statistics chosen for maximum power against specified alternatives (sometimes but not always giving the same ordering as (a) and/or (b), as I think RobertF was getting at): 'The Emperor's new tests', Perlman & Wu (1999), together with the comments & rejoinder, is very interesting (though difficult).

If you read the paper by Christensen that Zag linked to, you will see that in the first example he writes "With only this information, one must use the density itself to determine which data values seem weird and which do not". The clear implication is that with more information you needn't necessarily use the density itself to determine which data values seem weird and which do not.

In response to @whuber's comment ...

The likelihood ratio test is in fact a good example of Defn 2's being used. The p-value in this case is just the probability (under the null) of the the likelihood ratio's being larger or equal to that observed.

As an elementary example, you can test two hypotheses for the probability of success in a Bernoulli trial : $$H_0: \theta = 0.55$$ $$H_1: \theta = 0.35$$

Nine independent trials give $t$ successes:

$$\newcommand{\pr}{\mathrm{Pr}}\begin{array}{cccc} t & \pr(t|H_0) & \pr(t|H_1) & \frac{\pr(t|H_1)}{\pr(t|H_0)}=x\\ 0 & 0.00076 & 0.02071 & 27.372\\ 1 & 0.00832 & 0.10037 & 12.060\\ 2 & 0.04069 & 0.21619 & 5.3128\\ 3 & 0.11605 & 0.27162 & 2.3406\\ 4 & 0.21276 & 0.21939 & 1.0312\\ 5 & 0.26004 & 0.11813 & 0.4543\\ 6 & 0.21188 & 0.04241 & 0.2001\\ 7 & 0.11099 & 0.00979 & 0.0882\\ 8 & 0.03391 & 0.00132 & 0.0389\\ 9 & 0.00461 & 0.00008 & 0.0171 \end{array}$$

The likelihood ratio $x$ is your test statistic.

Using Defn 1 to get a p-value, you have to add up all the probabilities (under the null) for less (or equally) probabable values of $x$ than that observed. So, observing $t = 2$, you'd add up those for $2$, $8$, $1$, $9$, & $0$ successes to give $0.04069 + 0.03391 + 0.00832 + 0.00461 + 0.00076 = 0.08829$

Using Defn 2, you add up all the probabilities (under the null) for values of $x$ larger than (or equal to) that observed. So, observing $t = 2$, $x$ is larger for $0$ & $1$ successes so you add their probability under the null to that of $2$ successes to give a p-value of $0.04069 + 0.00832 + 0.00076 = 0.04977$.

It's clear that the latter procedure is the likelihood ratio test as usually understood, & that defined by the Neyman–Pearson lemma.