# p-value under composite null hypothesis

It is easy to evaluate p-value when the null hypothesis is simple $$(H_0: \theta = \theta_0)$$. Wikipedia gives the following formulas for this case:

Consider an observed test-statistic $$t$$ from unknown distribution $$T$$. Then the p-value $$p$$ is what the prior probability would be of observing a test-statistic value at least as "extreme" as $$t$$ if null hypothesis $$H_0$$ were true. That is:

• $$p = \Pr(T \geq t \mid H_0)$$ for a one-sided right-tail test,
• $$p = \Pr(T \leq t \mid H_0)$$ for a one-sided left-tail test,
• $$p = 2\min\{\Pr(T \geq t \mid H_0),\Pr(T \leq t \mid H_0)\}$$ for a two-sided test. If distribution $$T$$ is symmetric about zero, then $$p =\Pr(|T| \geq |t| \mid H_0)$$.

Did I understand correctly that the only thing we need to do to generalize these formulas to the composite null case $$(H_0: \theta \in \Theta_0)$$ is to add $$\displaystyle \sup_{\theta \in \Theta_0}$$? In other words, are the following statements true (below $$R$$ is a rejection region)?

1. if $$R = \{\mathbf{x}: T(\mathbf{x}) \ge c\}$$ then $$\displaystyle p(\mathbf{x}) = \sup_{\theta \in \Theta_0} \mathrm{Pr}_\theta(T(\mathbf{X}) \ge T(\mathbf{x}));$$
2. if $$R = \{\mathbf{x}: T(\mathbf{x}) \le c\}$$ then $$\displaystyle p(\mathbf{x}) = \sup_{\theta \in \Theta_0} \mathrm{Pr}_\theta(T(\mathbf{X}) \le T(\mathbf{x}));$$
3. if $$R = \{\mathbf{x}: |T(\mathbf{x})| \ge c\}$$ and null distribution of $$T(\mathbf{X})$$ is symmetric about zero, then $$\displaystyle p(\mathbf{x}) = \sup_{\theta \in \Theta_0} \mathrm{Pr}_\theta(|T(\mathbf{X})| \ge |T(\mathbf{x})|) = 2\cdot \sup_{\theta \in \Theta_0} \mathrm{Pr}_\theta(T(\mathbf{X}) \le -|T(\mathbf{x})|);$$
4. if $$R = \{\mathbf{x}: T(\mathbf{x}) \le c_1 ~ \text{or}~ T(\mathbf{x}) \ge c_2\}$$, where $$c_1 \lt c_2$$, then $$\displaystyle p(\mathbf{x}) = 2 \cdot \min\Big\{\sup_{\theta \in \Theta_0} \mathrm{Pr}_\theta(T(\mathbf{X}) \ge T(\mathbf{x})),~ \sup_{\theta \in \Theta_0} \mathrm{Pr}_\theta(T(\mathbf{X}) \le T(\mathbf{x})) \Big\}$$.

Edit. Larry Wasserman in his book "All of statistics" on p.158 says that the statement 1. is true:

Next, this post says that the statement 2. is true.
And from Example 8.3.28 from Casella's book "Statistical inference" (2nd ed.) it follows that the statement 3. is just a special case of the statement 1. (we just need to use $$|T(\mathbf{X})|$$ instead of $$T(\mathbf{X})$$ and $$|T(\mathbf{x})|$$ instead of $$T(\mathbf{x})$$).
Thus, it remains to find out whether the statement 4. is true.

• (4) follows from (3) upon rewriting $R$ as $\{|T(X) - a_1| \leq c_3\}$ for some $a_1$ and $c_3$. Sep 6, 2021 at 21:12
• @user551504 In (3) we have requirement that null distribution of $T(\mathbf{X})$ is symmetric about zero. So (4) will follow from (3) only if null distribution of $T(\mathbf{X}) - a_1$ is symmetric about zero. But we have no such requirement in (4). Also in your expression for $R$ there should be $\ge$ instead of $\le$. Sep 7, 2021 at 10:07
• Just choose $a_1$ so that its symmetric about zero Sep 7, 2021 at 16:03
• @user551504 It's not possible for asymmetric distributions like chi-square distribution. Sep 7, 2021 at 16:47
• It's just an algebraic manipulation, the set $c_1 < T(X) < c_2$ is equal to the set $|T(X) - \frac{c_1+c_2}{2}| < \frac{c_2-c_1}{2}$. These are the values of $a_1$ and $c_3$ I was referring to earlier. Then the rejection region is the complement of this set. Upon relabeling $T(X) - \frac{c_1+c_2}{2}$ as the test statistic, we're in case (3). Sep 7, 2021 at 16:50

There's a bit of confusion in the way that the results are stated, so we'll start by clarifying those. (Apologies, I engaged earlier without reading your question closely enough.) Define the $$p$$ value to be $$p(x) = \inf_{x \in \mathcal{R}_\alpha} \alpha$$ for some observed data $$x$$. Throughout we will use the notation that $$t=T(x)$$ is the observed statistic.

1. Choose a rejection region $$\mathcal{R}_\alpha = \{X : |T(X)| > c_\alpha\}$$ so that $$\sup_{\theta_0 \in \Theta_0} \mathbb{P}_{\theta_0} \left[X \in \mathcal{R}_\alpha\right] = \alpha$$. (Note, this precludes some discrete data distributions, we ignore that complication.) Whenever the rejection cutoff $$c_\alpha$$ is a decreasing function of $$\alpha$$, the $$p$$ value $$p(x) = \sup_{\theta_0 \in \Theta_0} \mathbb{P}_{\theta_0} \left[ |T(X)| > |t| \right]$$.

This follows almost immediately from the definitions. The $$p$$ value by definition equals $$p(x) = \inf_{\alpha: \, |t| > c_\alpha} \sup_{\theta_0 \in \Theta_0} \mathbb{P}_{\theta_0} \left[ |T(X)| > c_\alpha \right].$$ By the premise, the infimum is achieved at the upper bound $$c_\alpha = |t|$$ so that the result follows.

As a corollary, note that the premise holds when $$\Theta_0 = \{\theta_0\}$$ is a singleton and $$T(X)$$ is symmetric around zero under $$\theta_0$$. Drawing a picture makes this very clear.

1. Choose a rejection region $$\mathcal{R}_\alpha = \{X : T(X) < c_{1,\alpha} \text{ or } T(X) > c_{2,\alpha}\}$$ so that $$\sup_{\theta_0 \in \Theta_0} \mathbb{P}_{\theta_0} \left[X \in \mathcal{R}_\alpha\right] = \alpha$$. Further assume that the cutoffs are chosen so that $$\sup_\alpha c_{1, \alpha} = \inf_\alpha c_{2,\alpha}$$, making each observed test statistic $$T(x)$$ satisfy either exactly one of $$t < c_{1, \alpha}$$ or $$t > c_{2,\alpha}$$ for some $$\alpha$$. Whenever the cutoff $$c_{1,\alpha}$$ (respectively $$c_{2,\alpha}$$) is an increasing (respectively decreasing) function of $$\alpha$$, the $$p$$ value equals $$\min\{\sup_{\theta_0 \in \Theta_0} \mathbb{P}_{\theta_0} [T(X) < t \text{ or } T(X) > \tilde{c}_2], \sup_{\theta_0 \in \Theta_0} \mathbb{P}_{\theta_0} [T(X) < \tilde{c}_1 \text{ or } T(X) > t]\},$$ where $$\tilde{c}_1$$ corresponds with $$c_{\alpha, 2} = t$$, and likewise $$\tilde{c}_2$$ corresponds with $$c_{\alpha, 1} = t$$.

This can be routinely worked out using the same arguments as for (3). I encourage you to try the calculation.

As a corollary, when $$\Theta_0$$ is a singleton, $$\mathcal{R}_\alpha$$ is chosen to be equitailed, and the rejection cutoffs are monotonic, the expression for the $$p$$ value simplifies to $$\min\{2\mathbb{P}_{\theta_0} [T(X) < t], 2 \mathbb{P}_{\theta_0} [T(X) > t]\}.$$

• Well, thanks! As I understood, we got statement (4), i.e. $\displaystyle \text{p-value}(\mathbf{x}) = 2 \cdot \min\Big\{\sup_{\theta \in \Theta_0} \mathrm{Pr}_\theta(T(\mathbf{X}) \ge T(\mathbf{x})),~ \sup_{\theta \in \Theta_0} \mathrm{Pr}_\theta(T(\mathbf{X}) \le T(\mathbf{x})) \Big\}$ when the critical region $R_\alpha = \{\mathbf{x}: T(\mathbf{x}) \le c_{1,\alpha} \text{ or } T(\mathbf{x}) \ge c_{2,\alpha}\}$ is equitailed So I want to clarify – what exactly mean phrase "$R_\alpha$ is equitailed"? Sep 8, 2021 at 19:33
• Does this mean that $\mathrm{Pr}_{\theta}(T(\mathbf{X}) \le c_{1,\alpha}) = \mathrm{Pr}_{\theta}(T(\mathbf{X}) \ge c_{2,\alpha}), ~\forall \theta \in \Theta_0$ ? Sep 8, 2021 at 19:33
• Sorry, I don't currently have more time to spend on this. Moving forward, I would recommend you try to understand the basic results at the top of each "section". Then, if you want a more specialized result, state clear assumptions and try to simplify. Sep 8, 2021 at 19:44