# Incoherence of $p$-values

How do proponents of $$p$$-values as measures of evidence (a la Fisher) counter the incoherence arguments of Schervish etc?

In this setting, coherence means that if $$A$$ implies $$B$$, the evidence for $$B$$ should be at least as big as that for $$A$$.

Do they reject coherence as an essential property of evidence? Do they reject the examples showing $$p$$ violates coherence?

• P-value does not provide evidence for an alternative, but evidence against a null hypothesis. With his formulation, he defines the p-value in a way to set up an apparent paradox (creating the one-side p-value vs. two-sided p-value). It can be made coherent by defining the p-value over a set as the maximum over all the p-values for sets contained in it. In the specific case of $A=\{0\}$ and $B=(-\infty,0)$ with $x>0$, the p-value for $A$ would be equal to the (two-sided) p-value for $B$. Mar 18 at 18:29
• @john thanks, if you can, please do expand into an answer. Mar 19 at 1:06

P-value does not provide evidence for an alternative, but measures evidence against a null hypothesis.

In the Schervish article formulation, he defines the p-value in a way to set up an apparent paradox (creating the one-side p-value vs. two-sided p-value). He cites an example where you would fail to reject $$H_0:\mu=0$$ because the two-sided p-value is 0.074 but the one-sided p-value for testing $$H_0:\mu \le 0$$ is 0.037 so you would reject this null hypothesis. This seems to be a contradiction because on the one hand you say you can conclude $$\mu$$ is not equal to $$0$$ or any number less than $$0$$, but in the other case you cannot even say that it is not $$0$$ using the same data.

It can be made coherent by defining the p-value over a set as the maximum over all the p-values for sets contained in it. Take the specific case of $$A=\{0\}$$ and $$B=(−∞,0)$$ with $$\hat{\mu}=1.79$$ and standard error $$S.E.\{\sigma\}=1$$. The two-sided p-value for testing $$H_0:\mu=0$$ would be equal to $$2 \times (1-\Phi(1.79))\approx0.074$$. The two-sided p-value for testing $$H_0:\mu=-2$$ would be equal to $$2 \times (1-\Phi(1.79-(-2)))\approx0.0001$$. The p-value for B would be $$\max_{\mu_0\in B}\{2 \times (1-\Phi(1.79-\mu_0))\}=0.074$$ This is the approach of only considering two-sided p-values.

Alternatively, it can be made coherent by only considering one-sided tests and one-sided p-values which leads to the "three decision rule" attributed to Neyman. There is a detailed explanation here: Freedman, Laurence S. "An analysis of the controversy over classical one-sided tests." Clinical Trials 5.6 (2008): 635-640.

In almost all common hypothesis testing situations, p-values are:

1. monotonic. When testing a fixed null hypothesis, what naturally seems like more evidence against the null hypothesis leads to a smaller p-value.
2. coherent. For a fixed set of data, when testing two different null hypothesis with the same logic and where one null hypothesis is nested within the other, then the p-values will be in the logical direction expected.
a. For one-sided tests, the p-value for testing $$H_0:\mu \le \mu_1$$ will be smaller than or equal to the p-value for testing $$H_0:\mu \le \mu_2$$ if $$\mu_1<\mu_2$$. That is if the second hypothesis is rejected, then so will the first one.
b. For two-sided tests, suppose there are two points in the null hypothesis $$H_0:\mu=\mu_1 \text{ or } \mu=\mu_2$$, this is a intersection-union test scenario so the test should be based on the intersection of the rejection regions of the two scenarios $$H_0:\mu=\mu_1$$ and $$H_0:\mu=\mu_2$$. That means the p-value for testing this should be the maximum of the two p-values for the simpler null hypotheses. The same idea applies to generalize to the situation of testing $$H_0:\mu\le 0$$.
Berger, Roger L. "Multiparameter hypothesis testing and acceptance sampling." Technometrics 24.4 (1982): 295-300.