13
$\begingroup$

I am currently reading the paper

Duncan J Murdoch, Yu-Ling Tsai & James Adcock (2008) P-Values are Random Variables, The American Statistician, 62:3, 242-245, DOI: 10.1198/000313008X332421

In this paper, the authors argue that a p-value is itself a random test statistic. Moreover, given some test statistic $T$ that takes in an i.i.d. sample $\mathbf{X}$ and outputs a real number, a p-value is the probability integral transform of $T(\mathbf{X})$. That is, if $T(\mathbf{X})$ has a cumulative distribution function $F_{\tau}$, then the corresponding p-value is $F_{\tau}(T(\mathbf{X}))$. We can then decide whether to reject a null hypothesis based on this p-value. For example, a decision rule could be to reject the null hypothesis if the corresponding p-value is less than 0.05.

However, because $F_{\tau}$ is monotone increasing, I am not sure why we need to compute a p-value in the first place to decide whether or not to reject a null hypothesis. Can't the value of the test statistic $T(\mathbf{X})$ be used to decide? For example, if the decision rule is to reject the null hypothesis when the p-value is less than 0.05, then, if the inverse of $F_{\tau}$ exists, we can obtain the threshold value for $T(\mathbf{X})$ beyond which the null hypothesis is rejected. Furthermore, we should be able to compute type I and II error rates just using this threshold value for $T(\mathbf{X})$.

Improve this question
$\endgroup$

7 Answers 7

Reset to default
30
$\begingroup$

I think you are almost constructing p-values in the question.

You are correct, you can just set a threshold using $t=T(X)$, but as you point out, you'd like to calculate the error rate associated with that t-value. In order to do so you need to know the null distribution of your test statistic, which is $F$. So in order to find the t-value that has a 5% false positive under the null, you need to find $t$ such that $F(t)=0.05$, i.e. $t=F^{-1}(0.05)$

In many (most?) cases inverting the cdf is harder than evaluating it, hence it is much easier to calculate $p=F(T)$ and then check if $p<0.05$ than to check if $t<F^{-1}(0.05)$

Improve this answer
$\endgroup$
30
$\begingroup$

$p$-value is a random variable like the test statistic is a random variable, so it's not clear to me what it has to do with the rest of your question. Both are functions of random variables, hence random variables themselves.

Yes, you can use the threshold for the test statistic, but how would you choose it? How would you say that the threshold should be 1.96, 5, or 100? With $p$-values you have a familiar and easily interpretable scale of probabilities, that is not necessarily the case for test statistics.

Improve this answer
$\endgroup$
9
  • 3
    $\begingroup$ So does this mean that the only purpose of p-values is to transform the values of the test statistic for different tests into the same scale? i.e., to make values of the test statistic interpretable? $\endgroup$
    – mhdadk
    May 7, 2022 at 22:31
  • 14
    $\begingroup$ @mhdadk: Kind of; it would be more accurate to say that the only purpose of the test statistic is to get the p-value. $\endgroup$
    – Ben
    May 8, 2022 at 5:36
  • $\begingroup$ misleading - fixed p from different tests (especialyl of different sample size) in no way implies the same level of evidence, see e.g. Jeffreys-Lindley paradox $\endgroup$
    – innisfree
    May 10, 2022 at 7:42
  • 1
    $\begingroup$ @innisfree I never said anything about the level of evidence, I only said that they are easier to interpret. $\endgroup$
    – Tim ♦
    May 10, 2022 at 8:10
  • 1
    $\begingroup$ the edit you made has improved it a bit imho, let's agree to disagree $\endgroup$
    – innisfree
    May 10, 2022 at 13:22
16
$\begingroup$

I am not sure why we need to compute a p-value in the first place to decide whether or not to reject a null hypothesis.

We do not need to compute a p-value for hypothesis testing. We can do indeed what you suggest and compute a threshold value. An example is a likelihood ratio test as described by Neyman and Pearson. The example below from wikipedia uses a threshold value $\eta$ for the likelihood ratio

Define the rejection region of the null hypothesis for the Neyman–Pearson (NP) test as $$R_{NP} = \left\{ x: \frac{\mathcal{L}(\theta_0|x)}{\mathcal{L}(\theta_1|x)} \leq \eta \right\}$$ where $\eta$ is chosen so that $P( R_{NP} | \theta_0 ) = \alpha$ .

Why still compute the p-values?

Because it is not all about hypothesis testing.

For instance, often you want to provide a more detailed value than just a 'pass' or 'not-pass'.

Improve this answer
$\endgroup$
4
  • 4
    $\begingroup$ "often you want to provide a more detailed value than just a 'pass' or 'not-pass'" -- this is so, even without trying to interpret a p-value; consider for example, that a p-value allows someone with a preference for a different $\alpha$ than you used to tell whether they'd reject, without having to do a lot of additional computation. $\endgroup$
    – Glen_b
    May 9, 2022 at 2:56
  • $\begingroup$ Neyman and Pearson would agree up until the the "more detailed value" part. They were very against p-values as far as I know. www2.stat.duke.edu/~berger/papers/02-01.pdf $\endgroup$
    – Casey
    May 11, 2022 at 22:34
  • 1
    $\begingroup$ @Casey I don't need Neyman and Pearson to agree. :-) $\endgroup$
    – Sextus Empiricus
    May 11, 2022 at 23:04
  • $\begingroup$ @SextusEmpiricus Fair Enough :) I just think it's interesting that they stopped right before that point, and in some ways do consider the test statistic sufficient. $\endgroup$
    – Casey
    May 11, 2022 at 23:36
6
$\begingroup$

Yes, it's quite common for a threshold value to be found, and the test statistic be compared to it. It's especially common with the normal distribution; since this distribution is used so much, it's easy to look up what the threshold values for $0.05$ (for a single tailed test with $\alpha = 0.05$) and $0.025$ (for double-tailed $\alpha = 0.05$) are.

Calculating the p-value does give us more information than just finding a binary "Is it larger than the threshold?", which can be an advantage or a disadvantage (for instance, one disadvantage to giving p-values is that is raises the temptation of simply multiplying p-values of individual tests to get a composite p-value, which is fallacious reasoning).

Furthermore, we should be able to compute type I and II error rates just using this threshold value for T(X).

Type I error rate is just the maximum p-value under which we reject the null (at least, for continuous distributions). This is usually chosen directly, and p-value and threshold follow, rather than error rate being calculated from a free choice of threshold. Type II error rate is more complicated, and requires some alternative hypothesis being specified.

Improve this answer
$\endgroup$
6
$\begingroup$

One advantage I don't see mentioned yet is that p-values standardize different tests onto the same scale, which is a human-interpretable one.

If you have different tests $1,2,3$ with test statistics $T_1,T_2,T_3$ coming from null CDFs $F_1,F_2,F_3$, you can't just report $T_i$ to someone, you have to tell them $F_i$ so they can interpret it. But if we take $p_i = F_i(T_i)$, then all of $p_1,p_2,p_3$ are on a standardized scale. You can report $p_i$ without additional context and it's meaningful.

Even if you only wanted to report "pass/not pass" rather than the test statistic itself, you probably want to communicate how strict your threshold is. A natural measure of strictness for any test is how often the test would pass under the null hypothesis, but that's a threshold in p-value space (e.g. 0.05).

Improve this answer
$\endgroup$
1
  • $\begingroup$ same comment as above, misleading - fixed p from different tests (especially of different sample size) in no way implies the same level of evidence, see e.g. Jeffreys-Lindley paradox $\endgroup$
    – innisfree
    May 10, 2022 at 9:13
6
$\begingroup$

You're absolutely right. To perform a test of a fixed size, we don't need p-values at all. We may just define a rejection region of that size in terms of the test-statistic.

Possible advantages of reporting p, though, include that

  • readers may (if they reader wish) interpret p as a measure of evidence a la Fisher
  • readers may bring their own error rate, $\alpha$, to compare with your p
  • it may be computationally easier to compute p and compare it with the desired error rate, than invert the desired error rate to find the boundaries of the critical region of the test-statistic
  • readers may perform further calculations with p, such as adjusting p for multiple comparisons or converting it to a bound on a Bayes factor etc
Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ +1 The use of p-values for multiple testing correction (by virtue of being a probability itself) I think is the most important argument. $\endgroup$
    – user11130854
    May 10, 2022 at 11:34
  • $\begingroup$ This is interesting, but couldn't you equivalently define a joint distribution of test statistics over each test to find a multi-dimensional critical region? Not saying it would be easy but you could still do it without appealing to p-values right? $\endgroup$
    – Casey
    May 11, 2022 at 22:36
2
$\begingroup$

As a counter example to the idea that we can just use the raw test statistics, let's consider the following imaginary report:

We tested whether there were differences across product colors and found $F_{3, 190} = 3.886$. We tested whether was a difference across font designs and found $F_{2, 8} = 8.649$.

Is there a good reason for making the reader go through one more step?

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.