I am trying to see if I understand the definition of $p$-value as used by Sir R. A. Fisher and the one used today by frequentist statisticians (not sure how to call it better).
$p$-value according to Sir R. A. Fisher
A $p$-value for a given realization of a test statistic is the probability of getting an equally extreme or a more extreme realization under repeated sampling, provided that the null hypothesis $H_0$ is true. Extremity is defined in terms of the density of the test statistic (a random variable) given $H_0$, with "more extreme" meaning "having lower density". Note that such a definition of $p$-value does not involve any reference to an alternative hypothesis. Based on my rudimentary knowledge of history of statistics, I suppose Fisher would be fine with such a definition. As expressed by Christensen (2005),
The $p$-value is the probability of seeing something as weird or weirder than you actually saw. (There is no need to specify that it is computed under the null hypothesis because there is only one hypothesis.)
The key fact in a Fisherian test is that it makes no reference to any alternative hypothesis.
Also (square brackets added by me as deemphasis):
[First, $F$ tests and $\chi^2$ tests are typically rejected only for large values of the test statistic. Clearly, in Fisherian testing, that is inappropriate.] Finding the $p$ value for an $F$ test should involve ﬁnding the density associated with the observed $F$ statistic and ﬁnding the probability of getting any value with a lower density. [This will be a two-tailed test, rejecting for values that are very large or very close to 0.]
$p$-value used by frequentist statisticians today
I think the definition above (i.e. my attempt at rephrasing Fisher's definition) is not what is normally used today, because every now and then I encounter "one-sided" vs. "two-sided" $p$-values where extremity is clearly defined with reference to a specific alternative hypothesis. The low-density areas on the "uninteresting" side of $H_0$ are thus not counted. Otherwise the definition is the same.
- Does my understanding of $p$-value according to Sir R. A. Fisher make sense?
(Or did I overlook something important or made some mistakes in my explanation?)
- Is my understanding of today's definition of the $p$-value correct?
- Christensen, R. (2005). Testing Fisher, Neyman, Pearson, and Bayes. The American Statistician, 59(2), 121-126.