1
$\begingroup$

How do proponents of $p$-values as measures of evidence (a la Fisher) counter the incoherence arguments of Schervish etc?

In this setting, coherence means that if $A$ implies $B$, the evidence for $B$ should be at least as big as that for $A$.

Do they reject coherence as an essential property of evidence? Do they reject the examples showing $p$ violates coherence?

$\endgroup$
2
  • 1
    $\begingroup$ P-value does not provide evidence for an alternative, but evidence against a null hypothesis. With his formulation, he defines the p-value in a way to set up an apparent paradox (creating the one-side p-value vs. two-sided p-value). It can be made coherent by defining the p-value over a set as the maximum over all the p-values for sets contained in it. In the specific case of $A=\{0\}$ and $B=(-\infty,0)$ with $x>0$, the p-value for $A$ would be equal to the (two-sided) p-value for $B$. $\endgroup$
    – John L
    Mar 18 at 18:29
  • $\begingroup$ @john thanks, if you can, please do expand into an answer. $\endgroup$
    – innisfree
    Mar 19 at 1:06
2
$\begingroup$

P-value does not provide evidence for an alternative, but measures evidence against a null hypothesis.

In the Schervish article formulation, he defines the p-value in a way to set up an apparent paradox (creating the one-side p-value vs. two-sided p-value). He cites an example where you would fail to reject $H_0:\mu=0$ because the two-sided p-value is 0.074 but the one-sided p-value for testing $H_0:\mu \le 0$ is 0.037 so you would reject this null hypothesis. This seems to be a contradiction because on the one hand you say you can conclude $\mu$ is not equal to $0$ or any number less than $0$, but in the other case you cannot even say that it is not $0$ using the same data.

It can be made coherent by defining the p-value over a set as the maximum over all the p-values for sets contained in it. Take the specific case of $A=\{0\}$ and $B=(−∞,0)$ with $\hat{\mu}=1.79$ and standard error $S.E.\{\sigma\}=1$. The two-sided p-value for testing $H_0:\mu=0$ would be equal to $2 \times (1-\Phi(1.79))\approx0.074$. The two-sided p-value for testing $H_0:\mu=-2$ would be equal to $2 \times (1-\Phi(1.79-(-2)))\approx0.0001$. The p-value for B would be $$\max_{\mu_0\in B}\{2 \times (1-\Phi(1.79-\mu_0))\}=0.074$$ This is the approach of only considering two-sided p-values.

Alternatively, it can be made coherent by only considering one-sided tests and one-sided p-values which leads to the "three decision rule" attributed to Neyman. There is a detailed explanation here: Freedman, Laurence S. "An analysis of the controversy over classical one-sided tests." Clinical Trials 5.6 (2008): 635-640.

In almost all common hypothesis testing situations, p-values are:

  1. monotonic. When testing a fixed null hypothesis, what naturally seems like more evidence against the null hypothesis leads to a smaller p-value.
  2. coherent. For a fixed set of data, when testing two different null hypothesis with the same logic and where one null hypothesis is nested within the other, then the p-values will be in the logical direction expected.
    a. For one-sided tests, the p-value for testing $H_0:\mu \le \mu_1$ will be smaller than or equal to the p-value for testing $H_0:\mu \le \mu_2$ if $\mu_1<\mu_2$. That is if the second hypothesis is rejected, then so will the first one.
    b. For two-sided tests, suppose there are two points in the null hypothesis $H_0:\mu=\mu_1 \text{ or } \mu=\mu_2$, this is a intersection-union test scenario so the test should be based on the intersection of the rejection regions of the two scenarios $H_0:\mu=\mu_1$ and $H_0:\mu=\mu_2$. That means the p-value for testing this should be the maximum of the two p-values for the simpler null hypotheses. The same idea applies to generalize to the situation of testing $H_0:\mu\le 0$.
    Berger, Roger L. "Multiparameter hypothesis testing and acceptance sampling." Technometrics 24.4 (1982): 295-300.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.