Is the following textbook definition of $p$-value correct?

Question

I have found the following definition of $p$-value in an introductory statistics textbook (not in English, so I am translating it):

$p$-value is the probability of getting a result that is at least as much in favour of $H_1$ as the observed result, provided that $H_0$ is correct.

Is this definition correct? If not, what exactly is wrong with it?

Answer 1

The issue I have with this is that as it stands it is not a definition, as long as there is no formal definition what "in favour of $H_1$" actually means. Furthermore, as you probably know, Fisher and others have defined tests and p-values without specifying a $H_1$.

Here's an attempt to make the "definition" correct. A test generally is defined by a test statistic $T$ and a "discrepancy" $d$ (see below), and a p-value is $P_{H_0}\{d(T,H_0)\ge d(t,H_0)\}$, where $d$ is a suitably defined discrepancy function between a value of the test statistic $T$ (where $t$ is the actual value observed in the data) and what is "expected" under the $H_0$.

One way of defining $T$ and $d$ is to set up an alternative $H_1$ and to choose $T$ and $d$ so that optimal rejection probability at any fixed level $\alpha$ is achieved under $H_1$. This is Neyman and Pearson's approach, and it may require side conditions such as the test being unbiased, because for example in the two-sided case otherwise uniform optimality under $H_1$ cannot be achieved.

Using the concept of unbiasedness, given $T$ and $d$ (which may or may not have been derived using a specific alternative), one can define an implicit (composite) alternative $H_1$ of any given test as all distributions $Q$ so that $Q\{d(T,H_0)\ge d(t,H_0)\}>P_{H_0}\{d(T,H_0)\ge d(t,H_0)\}$. I assume here that this can be fulfilled uniformly over all possible values of $t$ (probably it's good enough to relax this a bit by asking for "$\ge$" instead of "$>$", and "$>$" for at least one $t$ or something). Note that if we don't think about p-values but rather about $\alpha$-level testing for fixed $\alpha$, one can define an "implicit alternative" based on the critical value $t_\alpha$, which should always be possible; I haven't thought much about how much more restrictive the uniformity assumption is, but it seems to me that this is what is needed to make the definition in question valid.

Using this definition, it is simply the case that $t$ can be seen more "in favour of $H_1$" if $d(t,H_0)$ is larger, and this makes the definition in the question correct. (The issue with composite $H_1$ such as $H_1:\ \mu\neq \mu_0$ when testing $H_0:\ \mu=\mu_0$ is just to define $d$ accordingly, for example using $d(T,H_0)=|T-\mu_0|$ rather than $T-\mu_0$ (or maybe $d(T,H_0)=(T-\mu_0)1(T-\mu_0>0)$, $T$ here being an estimator of $\mu$, if we insist on a discrepancy being non-negative) for $H_1:\ \mu>\mu_0$.

See also my answer here.

Answer 2

That is the correct definition for a test with a simple null hypothesis. For a test with a composite null hypothesis (i.e., more than one possible parameter value in the null space) things are complicated a bit by the fact that the p-value is the supremum over the conditional probabilities over the parameters in the null space.

Answer 3

The more general definition of a p-value is

the p-value is the probability of getting a result that is at least as extreme as the observed result, provided that $H_0$ is correct.

The definition is not clear about what 'extreme' means. One example of a p-value is a p-value that defines the degree of extremeness as values that are more in favour of $H_1$. This gives the definition in your question

the p-value is the probability of getting a result that is at least as much in favour of $H_1$ as the observed result, provided that $H_0$ is correct

1. This is not the definition of a p-value but a definition of a p-value.

2. It is a bit difficult to see what they mean by

   at least as much in favour of $H_1$

   One could view this definition in terms of the likelihood ratio test which is (for simplicity we use simple hypotheses):

   $$P \left ( \frac{\mathcal{L}(H_1|X)}{\mathcal{L}(H_0|X)} \geq \frac{\mathcal{L}(H_1|x_{observed})}{\mathcal{L}(H_0|x_{observed})} \right)$$

   The $p$-value (in a likelihood ratio test) is the probability of getting a result for which the likelihood ratio of the hypotheses $H_1$ and $H_0$ is at least as much as the observed result, provided that $H_0$ is correct.

---

^{I call it not clear what they mean with 'at least as much in favour' because I had initially a different thought about it than the likelihood ratio}

^{- I would prefer to use phrasing in terms of that likelihood. The term 'at least as much in favour of $H_1$' confused me initially and made me think of the wrong $P \left ( {\mathcal{L}(H_1|X)}>{\mathcal{L}(H_1|x_{observed})} \right)$}

^{Example, say we have a sample $X \sim N(\mu,1)$ to test the hypotheses that are $H_0:\mu = 0$ and $H_1: \mu =2$. Let the observation be $x = 3$, then the values that are at least as much in favour for $H_1$ are in between $1$ and $3$ and the probability for that under $H_0$ is $\Phi(3)-\Phi(1) \approx 0.157$. But with the likelihood ratio test we would not consider the values between $1$ and $3$ that are more in favour of $H_1$ and instead we would consider the values $>3$ for which the outcome is relatively more in favour of $H_1$ in comparison to $H_0$.}

^{- The term 'in favour' also initially confused me because it implies that the observed result must be in favour of $H_1$ but that does not need to be the case. It can be that the values are in favour of $H_0$.}

^{Example, say we have a sample $X \sim N(\mu,1)$ to test the hypotheses that are $H_0:\mu = 0$ and $H_1: \mu =10$. Let the observation be $x = 3$, then this is a value that is not in favour of $H_1$ (at least not compared to $H_0$).}

Answer 4

It's obviously a translation, but logically correct. P-value is probability, under $H_0$, of a more extreme result of the test statistic in the direction(s) of the alternative hypothesis than the observed value of the test statistic. [For a two-sided alternative, two probabilities are added to get the P-value.]

Consider the following normal samples (from R) and a Welch 2-sample t test to see whether their sample means are significantly different; specifically to test $H_0: \mu_1 = \mu_2$ against $H_a: \mu_1 < \mu_2.$

set.seed(1234)
x1 = rnorm(20, 100, 10)
x2 = rnorm(25, 110, 12)

        Welch Two Sample t-test

data:  x1 and x2
t = -2.0301, df = 40.54, p-value = 0.02447
alternative hypothesis: 
 true difference in means is less than 0
95 percent confidence interval:
      -Inf -1.046619
sample estimates:
mean of x mean of y 
 97.49336 103.62038 

Under $H_0,$ the test statistic is approximately distributed as Student's t distribution with 41 degrees of freedom. So one would reject $H_0$ at the 5% level if $T \le -1.683.$

qt(.05, 41)
[1] -1.682878

However, the $T = -2.0301$ is even smaller than this 'critical value'. The P=value is the probability $P(T \le -2.0301) \approx 0.0244,$ computed under $H_0.$

pt(-2.0301, 41)
[1] 0.0244342

In the figure below the P-value is the area under the density curve to the left of the vertical red line.

By contrast, if this were a two-sided test $H_0: \mu_1=\mu_2$ against $H_a: \mu_1 \ne \mu_2,$ then the P-value would be $P(|T| \ge 2.0301) \approx 2(0.244) = 0.0488.$ So the sample means differ significantly at the 5% level of significance.

t.test(x1, x2)

        Welch Two Sample t-test

data:  x1 and x2
t = -2.0301, df = 40.54, p-value = 0.04894
alternative hypothesis: 
 true difference in means is not equal to 0
95 percent confidence interval:
 -12.22426952  -0.02977364
sample estimates:
mean of x mean of y 
 97.49336 103.62038 

In the figure below the P-value is the sum of the areas outside the vertical red lines.

Note: If this were a pooled two-sample t test, then the degrees of freedom for the t statistic under $H_0$ would be $\nu = n_1+n_2 - 2 = 43.$ Because this is a Welch t test and sample variances are not exactly equal, the degrees of freedom are computed according to a formula that involves $n_1, n_2, S_1^2,$ and $S_2^2.$ giving $\min(n_1-1,n_2-1) \le \nu \le n_1+n_2-2.$

For the current data, $\nu = 40.54.$ R shows fractional degrees of freedom; printed tables of t distributions and some software program use only integer degrees of freedom.