Our tag definition of the $p$-value says

In frequentist hypothesis testing, the $p$-value is the probability of a result as extreme (or more) than the observed result, under the assumption that the null hypothesis is true.

But how do we define what is more extreme? In "A Dialog Between a Teacher and a Thoughtful Student", @whuber shows that extreme can be defined w.r.t. the likelihood ratio under $H_0$ vs. $H_1$ (or $H_A$ in the original notation), $LR=\frac{P(data|H_1)}{P(data|H_0)}$. The larger the LR, the more extreme the result. So far so good.

In @whuber's example, $H_0$ is one-sided, and so is $H_1$. Then it is not so difficult to find which regions of values of the test statistic produce the largest LRs. Thus we have little conceptual trouble in finding the $p$-value; we just integrate the area under the null distribution for all possible values of the test statistic that are equally or more extreme (have equal or larger LR) than the observed value.

However, it is not so clear when $H_1$ is two sided, such as in a two-sided $t$-test. While the left tail of the null distribution would produce the largest LRs for an alternative to the left of the hypothesized value in $H_0$, the left tail would not at all be extreme for an alternative to the right of $H_0$; in fact, it would be the opposite tail that would be extreme. The problem is, both alternatives belong to $H_1$.

Q: How do we deal with such a situation? What is the principled way of defining extremeness when contradicting levels of LRs can arise under different instances within $H_1$?

P.S. I have earlier asked a related question of whether $p$-value ever depends on the alternative. I learned that using the modern (post Fisher) definition of the $p$-value, it does.

  • 1
    $\begingroup$ As with many questions about p-values, it can help if you start from a Neyman-Fisher type approach, where we have a rejection rule at some significance level (but not necessarily LRT*). Then "more extreme" is a consequence of specifying a choice of rejection regions. If you define a set of nested rejection regions (sets) $R_i$ at a sequence of increasing significance levels (so that $R_i\subset R_{i+1}$ for $\alpha_{i+1}>\alpha_i$. Then test statistics in $R_i$ are more extreme that test statistics that are in $R_{i+1}$ but not in $R_i$. ... ctd $\endgroup$ – Glen_b Aug 19 at 9:33
  • $\begingroup$ ctd .. * e.g. LRT is no use when you don't have a parametric distributional model. So, for example, if we do a two-sided two-sample permutation test of difference in means and specify our test statistic as $\bar{x}-\bar{y}$, then we might proceed to construct rejection regions by considering $|\bar{x}-\bar{y}|$ which gives us just such a set of nested rejection regions; and so "more extreme" $\bar{x}-\bar{y}$ is then "larger $|\bar{x}-\bar{y}|$". Other choices are possible, though. $\endgroup$ – Glen_b Aug 19 at 9:43
  • $\begingroup$ @Glen_b, thank you! I suppose the real question is, how do we define these rejection regions? How do we come up with a sensible rule for that? Also, does the essence of the problem change if we use LR vs. something else? I used the LR example as I found whuber's explanation quite clear. I think it would be helpful to stick to it for simplicity and for being able to relate to something that is already clear. $\endgroup$ – Richard Hardy Aug 19 at 9:53
  • $\begingroup$ While it seems that we're just "shifting the problem", we do actually make it easier (specifically, less abstract) by considering the rejection region directly in the context of our specific situation/working hypothesis - which usually tells us what's "more consistent" with H1 (& which might sometimes vary from application to application even with the same test statistic). I wouldn't choose to only write an answer in terms of likelihoods or ratios of them for the reason I already gave -- it doesn't deal with a large subset of real testing situations where we don't have a parametric model. $\endgroup$ – Glen_b Aug 19 at 10:07
  • $\begingroup$ @Glen_b, I see it differently than you, but I am open to alternative viewpoints as well. In your view, I suppose we would not want to exclude the case of likelihood ratios either, so that we do not exclude a large subset of real testing situations where we have a parametric model. But there is likely a trade-off between generality and clarity/intuition; I would go for the latter here if I could choose. We could take the basic two-sided $t$-test as an example; it probably does not get much simpler than that. Why do we use both tails symmetrically for defining the $p$-value there? $\endgroup$ – Richard Hardy Aug 19 at 11:26

In addition to scenarios in two-sided tests, this question arises in a less avoidable way in group sequential clinical trials.

In a group sequential trial there are a set of analysis times, and a stopping boundary specifying thresholds at each analysis for the trial to stop. In calculating $p$-values or confidence intervals it is necessary to specify a ordering of the possible outcomes. For example, if you stop at time 2 out of 4 with a $Z$-score of 3, how does that compare to stopping at time 3 with a $Z$-score of 2.5?

Among the orderings actually proposed are

  • ordering by the magnitude of difference
  • ordering by time, so that any stopping at an earlier time is more extreme than any stopping at a later time

These are genuine choices; different people could legitimately pick different orderings. Ordering by the magnitude of difference tends to lead to narrower confidence intervals, more accurate p-values, and less bias, but it increases the sensitivity of the analysis to the (unobservable) times at which future analyses of a stopped trial would have occurred.

(Reference: short course by Kittleson and Gillen)

| cite | improve this answer | |
  • $\begingroup$ Thank you for your answer! It seems to focus on sequential trials, but does it answer the rather basic question I have asked? E.g. in a two-sided t-test, the practice seems to be that both tails are considered extreme; I am wondering what the basis for that is. More generally, without specifying a prior, how do we know which instances within the alternative to pick for defining extremeness? As I argue in the OP, different instances yield very different regions of extreme values, so the choice is important. $\endgroup$ – Richard Hardy Sep 1 at 7:07

Defining extremeness of test statistic and defining p-value for a two-sided test...

I would suggest that an appropriate perspective here is that, when one has the "right" statistic, the statistic itself tells you what "extremeness" means for the test problem at hand---one-sided or two-sided. The more basic question is therefore what the "right" statistic is. Test problems are special cases of optimization problems---you want to maximize power subject to size constraint. So this means defining the "right" solution concept.

For example, finding the most powerful test for the test problem with a simple null vs. simple alternative is a special case of a linear program: $$ \sup_{0 \leq \phi \leq 1, \, \\ \\ \int \phi(\omega) f_0(\omega) d\mu \leq \alpha} \int \phi(\omega) f_1(\omega) d\mu. $$ It is a general fact that a solution $\phi^*$for any such program takes the form $$ \phi^* = \begin{cases} 1 & \text{if } f_1 \geq k f_0 \\ 0 & \text{if } f_1 \geq k f_0, \end{cases} $$ for some $k$. In the context of a test problem, a natural interpretation is then that one rejects when the likelihood ratio statistic $\frac{f_1}{f_0}$ is larger than $k$.

(It is suggested in the comments that the threshold $k$ is interpreted to be the "shadow price" of the size constraint. Apparently this terminology is borrowed from economics. $k$ is the Kuhn-Tucker-Lagrange multiplier of the problem. For interior solutions, typically one would say that if $\alpha$---the budget, in economic problems---is relaxed by $\epsilon$, the power of the test increases by $k \epsilon$. This interpretation, however, does not really hold for linear programs in general.)

Similarly, finding a most powerful test of composite null vs. simple alternative amounts to solving a linear program. The solution to the corresponding dual program tells us that the most powerful statistic is a likelihood ratio statistic with respect to the least favorable Bayesian prior on the null. (The simple null case is a special case, with trivial prior.)

Tests with one-sided alternatives for models with monotone likelihood ratio (MLR) property is of course another example. MLR means the model admits a ranking of likelihood ratios that's invariant with respect to data $\omega$. So the likelihood ratio test is a most powerful test, almost by assumption.

For two-sided alternatives, e.g. $\Gamma_0 = \{\gamma_0\}$ and $\Gamma_1 = (-\infty,\gamma_0)\cup (\gamma_0, \infty)$ for normal densities parametrized by mean $\gamma \in \mathbb{R}$, the most powerful test does not exist in general. Therefore the right statistic needs to be determined by some other criterion---e.g. one can instead look for a locally most powerful test.

A test $\phi^*$ is a locally most powerful test if for any other test $\phi$, there exists an open neighborhood $N_{\gamma_0, \phi}$ of the null hypothesis such that $\phi^*$ has uniformly higher power than $\phi$ on $N_{\gamma_0, \phi}$. The corresponding first-order optimality condition gives the criterion $$ \phi^* = \begin{cases} 1 & \text{if } \frac{\partial^2}{\partial \gamma^2}f_{\gamma_0} \geq k_1 \frac{\partial}{\partial \gamma} f_{\gamma_0} + k_2 f_{\gamma_0} \\ 0 & \text{if } \frac{\partial^2}{\partial \gamma^2}f_{\gamma_0} < k_1 \frac{\partial}{\partial \gamma} f_{\gamma_0} + k_2 f_{\gamma_0} \end{cases} $$ for some $k_1$ and $k_2$. Substituting the normal density into above expressions, we have that $\phi^*$ rejects when $|x- \gamma_0|$ is large---a two-sided test.

| cite | improve this answer | |
  • $\begingroup$ Thank you for your answer! It is is more technical than I am comfortable with, but I will try to digest it. I would also love to get some more intuition. $\endgroup$ – Richard Hardy Sep 10 at 6:09
  • $\begingroup$ @RichardHardy The FOC for two-sided $\phi^*$ probably has an interpretation---"if the likelihood under the null is sufficiently convex with respect to parameter, you reject the null". There is a Fisher-information flavor to the condition. $\endgroup$ – Michael Sep 10 at 6:15
  • $\begingroup$ @RichardHardy Also, if we extend the Neyman-Pearson setting to a null with two elements $f_{0,1}$ and $f_{0,2}$, then the LR test extends to a statistic that rejects when $f_1$ is larger than $k_1 f_{0,1} + k_2 f_{0,2}$ for some $k_1$ and $k_2$. Perhaps the shadow price interpretation is useful here(?). $\endgroup$ – Michael Sep 10 at 7:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.