2
$\begingroup$

It is easy to evaluate p-value when the null hypothesis is simple $(H_0: \theta = \theta_0)$. Wikipedia gives the following formulas for this case:

Consider an observed test-statistic $t$ from unknown distribution $T$. Then the p-value $p$ is what the prior probability would be of observing a test-statistic value at least as "extreme" as $t$ if null hypothesis $H_0$ were true. That is:

  • $p = \Pr(T \geq t \mid H_0)$ for a one-sided right-tail test,
  • $p = \Pr(T \leq t \mid H_0)$ for a one-sided left-tail test,
  • $p = 2\min\{\Pr(T \geq t \mid H_0),\Pr(T \leq t \mid H_0)\}$ for a two-sided test. If distribution $T$ is symmetric about zero, then $p =\Pr(|T| \geq |t| \mid H_0)$.

Did I understand correctly that the only thing we need to do to generalize these formulas to the composite null case $(H_0: \theta \in \Theta_0)$ is to add $\displaystyle \sup_{\theta \in \Theta_0}$? In other words, are the following statements true (below $R$ is a rejection region)?

  1. if $R = \{\mathbf{x}: T(\mathbf{x}) \ge c\}$ then $\displaystyle p(\mathbf{x}) = \sup_{\theta \in \Theta_0} \mathrm{Pr}_\theta(T(\mathbf{X}) \ge T(\mathbf{x}));$
  2. if $R = \{\mathbf{x}: T(\mathbf{x}) \le c\}$ then $\displaystyle p(\mathbf{x}) = \sup_{\theta \in \Theta_0} \mathrm{Pr}_\theta(T(\mathbf{X}) \le T(\mathbf{x}));$
  3. if $R = \{\mathbf{x}: |T(\mathbf{x})| \ge c\}$ and null distribution of $T(\mathbf{X})$ is symmetric about zero, then $\displaystyle p(\mathbf{x}) = \sup_{\theta \in \Theta_0} \mathrm{Pr}_\theta(|T(\mathbf{X})| \ge |T(\mathbf{x})|) = 2\cdot \sup_{\theta \in \Theta_0} \mathrm{Pr}_\theta(T(\mathbf{X}) \le -|T(\mathbf{x})|);$
  4. if $R = \{\mathbf{x}: T(\mathbf{x}) \le c_1 ~ \text{or}~ T(\mathbf{x}) \ge c_2\}$, where $c_1 \lt c_2$, then $\displaystyle p(\mathbf{x}) = 2 \cdot \min\Big\{\sup_{\theta \in \Theta_0} \mathrm{Pr}_\theta(T(\mathbf{X}) \ge T(\mathbf{x})),~ \sup_{\theta \in \Theta_0} \mathrm{Pr}_\theta(T(\mathbf{X}) \le T(\mathbf{x})) \Big\}$.

Edit. Larry Wasserman in his book "All of statistics" on p.158 says that the statement 1. is true: enter image description here

Next, this post says that the statement 2. is true.
And from Example 8.3.28 from Casella's book "Statistical inference" (2nd ed.) it follows that the statement 3. is just a special case of the statement 1. (we just need to use $|T(\mathbf{X})|$ instead of $T(\mathbf{X})$ and $|T(\mathbf{x})|$ instead of $T(\mathbf{x})$).
Thus, it remains to find out whether the statement 4. is true.

$\endgroup$
7
  • $\begingroup$ (4) follows from (3) upon rewriting $R$ as $\{|T(X) - a_1| \leq c_3\}$ for some $a_1$ and $c_3$. $\endgroup$
    – user551504
    Sep 6, 2021 at 21:12
  • $\begingroup$ @user551504 In (3) we have requirement that null distribution of $T(\mathbf{X})$ is symmetric about zero. So (4) will follow from (3) only if null distribution of $T(\mathbf{X}) - a_1$ is symmetric about zero. But we have no such requirement in (4). Also in your expression for $R$ there should be $\ge$ instead of $\le$. $\endgroup$
    – Rodvi
    Sep 7, 2021 at 10:07
  • $\begingroup$ Just choose $a_1$ so that its symmetric about zero $\endgroup$
    – user551504
    Sep 7, 2021 at 16:03
  • $\begingroup$ @user551504 It's not possible for asymmetric distributions like chi-square distribution. $\endgroup$
    – Rodvi
    Sep 7, 2021 at 16:47
  • $\begingroup$ It's just an algebraic manipulation, the set $c_1 < T(X) < c_2$ is equal to the set $|T(X) - \frac{c_1+c_2}{2}| < \frac{c_2-c_1}{2}$. These are the values of $a_1$ and $c_3$ I was referring to earlier. Then the rejection region is the complement of this set. Upon relabeling $T(X) - \frac{c_1+c_2}{2}$ as the test statistic, we're in case (3). $\endgroup$
    – user551504
    Sep 7, 2021 at 16:50

1 Answer 1

2
$\begingroup$

There's a bit of confusion in the way that the results are stated, so we'll start by clarifying those. (Apologies, I engaged earlier without reading your question closely enough.) Define the $p$ value to be $p(x) = \inf_{x \in \mathcal{R}_\alpha} \alpha$ for some observed data $x$. Throughout we will use the notation that $t=T(x)$ is the observed statistic.


  1. Choose a rejection region $\mathcal{R}_\alpha = \{X : |T(X)| > c_\alpha\}$ so that $\sup_{\theta_0 \in \Theta_0} \mathbb{P}_{\theta_0} \left[X \in \mathcal{R}_\alpha\right] = \alpha$. (Note, this precludes some discrete data distributions, we ignore that complication.) Whenever the rejection cutoff $c_\alpha$ is a decreasing function of $\alpha$, the $p$ value $p(x) = \sup_{\theta_0 \in \Theta_0} \mathbb{P}_{\theta_0} \left[ |T(X)| > |t| \right]$.

This follows almost immediately from the definitions. The $p$ value by definition equals $$p(x) = \inf_{\alpha: \, |t| > c_\alpha} \sup_{\theta_0 \in \Theta_0} \mathbb{P}_{\theta_0} \left[ |T(X)| > c_\alpha \right].$$ By the premise, the infimum is achieved at the upper bound $c_\alpha = |t|$ so that the result follows.

As a corollary, note that the premise holds when $\Theta_0 = \{\theta_0\}$ is a singleton and $T(X)$ is symmetric around zero under $\theta_0$. Drawing a picture makes this very clear.


  1. Choose a rejection region $\mathcal{R}_\alpha = \{X : T(X) < c_{1,\alpha} \text{ or } T(X) > c_{2,\alpha}\}$ so that $\sup_{\theta_0 \in \Theta_0} \mathbb{P}_{\theta_0} \left[X \in \mathcal{R}_\alpha\right] = \alpha$. Further assume that the cutoffs are chosen so that $\sup_\alpha c_{1, \alpha} = \inf_\alpha c_{2,\alpha}$, making each observed test statistic $T(x)$ satisfy either exactly one of $t < c_{1, \alpha}$ or $t > c_{2,\alpha}$ for some $\alpha$. Whenever the cutoff $c_{1,\alpha}$ (respectively $c_{2,\alpha}$) is an increasing (respectively decreasing) function of $\alpha$, the $p$ value equals $$\min\{\sup_{\theta_0 \in \Theta_0} \mathbb{P}_{\theta_0} [T(X) < t \text{ or } T(X) > \tilde{c}_2], \sup_{\theta_0 \in \Theta_0} \mathbb{P}_{\theta_0} [T(X) < \tilde{c}_1 \text{ or } T(X) > t]\},$$ where $\tilde{c}_1$ corresponds with $c_{\alpha, 2} = t$, and likewise $\tilde{c}_2$ corresponds with $c_{\alpha, 1} = t$.

This can be routinely worked out using the same arguments as for (3). I encourage you to try the calculation.

As a corollary, when $\Theta_0$ is a singleton, $\mathcal{R}_\alpha$ is chosen to be equitailed, and the rejection cutoffs are monotonic, the expression for the $p$ value simplifies to $$\min\{2\mathbb{P}_{\theta_0} [T(X) < t], 2 \mathbb{P}_{\theta_0} [T(X) > t]\}.$$

$\endgroup$
3
  • $\begingroup$ Well, thanks! As I understood, we got statement (4), i.e. $\displaystyle \text{p-value}(\mathbf{x}) = 2 \cdot \min\Big\{\sup_{\theta \in \Theta_0} \mathrm{Pr}_\theta(T(\mathbf{X}) \ge T(\mathbf{x})),~ \sup_{\theta \in \Theta_0} \mathrm{Pr}_\theta(T(\mathbf{X}) \le T(\mathbf{x})) \Big\}$ when the critical region $R_\alpha = \{\mathbf{x}: T(\mathbf{x}) \le c_{1,\alpha} \text{ or } T(\mathbf{x}) \ge c_{2,\alpha}\}$ is equitailed So I want to clarify – what exactly mean phrase "$R_\alpha$ is equitailed"? $\endgroup$
    – Rodvi
    Sep 8, 2021 at 19:33
  • $\begingroup$ Does this mean that $\mathrm{Pr}_{\theta}(T(\mathbf{X}) \le c_{1,\alpha}) = \mathrm{Pr}_{\theta}(T(\mathbf{X}) \ge c_{2,\alpha}), ~\forall \theta \in \Theta_0$ ? $\endgroup$
    – Rodvi
    Sep 8, 2021 at 19:33
  • $\begingroup$ Sorry, I don't currently have more time to spend on this. Moving forward, I would recommend you try to understand the basic results at the top of each "section". Then, if you want a more specialized result, state clear assumptions and try to simplify. $\endgroup$
    – user551504
    Sep 8, 2021 at 19:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.