4
$\begingroup$

I am trying to see if I understand the definition of $p$-value as used by Sir R. A. Fisher and the one used today by frequentist statisticians (not sure how to call it better).

$p$-value according to Sir R. A. Fisher
A $p$-value for a given realization of a test statistic is the probability of getting an equally extreme or a more extreme realization under repeated sampling, provided that the null hypothesis $H_0$ is true. Extremity is defined in terms of the density of the test statistic (a random variable) given $H_0$, with "more extreme" meaning "having lower density". Note that such a definition of $p$-value does not involve any reference to an alternative hypothesis. Based on my rudimentary knowledge of history of statistics, I suppose Fisher would be fine with such a definition. As expressed by Christensen (2005),

The $p$-value is the probability of seeing something as weird or weirder than you actually saw. (There is no need to specify that it is computed under the null hypothesis because there is only one hypothesis.)

Also there:

The key fact in a Fisherian test is that it makes no reference to any alternative hypothesis.

Also (square brackets added by me as deemphasis):

[First, $F$ tests and $\chi^2$ tests are typically rejected only for large values of the test statistic. Clearly, in Fisherian testing, that is inappropriate.] Finding the $p$ value for an $F$ test should involve finding the density associated with the observed $F$ statistic and finding the probability of getting any value with a lower density. [This will be a two-tailed test, rejecting for values that are very large or very close to 0.]

$p$-value used by frequentist statisticians today
I think the definition above (i.e. my attempt at rephrasing Fisher's definition) is not what is normally used today, because every now and then I encounter "one-sided" vs. "two-sided" $p$-values where extremity is clearly defined with reference to a specific alternative hypothesis. The low-density areas on the "uninteresting" side of $H_0$ are thus not counted. Otherwise the definition is the same.

Questions

  1. Does my understanding of $p$-value according to Sir R. A. Fisher make sense?
    (Or did I overlook something important or made some mistakes in my explanation?)
  2. Is my understanding of today's definition of the $p$-value correct?

References

Improve this question
$\endgroup$
8
  • 2
    $\begingroup$ I don't know if he would have used density per se. Imagine a test statistic whose distribution was bi-modal under the null (I can't think of one, but nothing prevents a test static from having a weird distribution). In that case, a "low density" could occur in the middle of the distribution, but my intuition is that Fisher wouldn't have the same interpretation there as in the tails. $\endgroup$
    – gung - Reinstate Monica
    Jan 9, 2019 at 18:03
  • $\begingroup$ @gung, well, that is exactly why density is useful - it allows us to deal with ugly distributions. If all densities were concave, we could go for critical values and intervals of points futher away from the centre than critical values, skipping references to density. This is not to say that Fisher should share my sentiment, of course, but your point goes against the third quote of Christensen (2005), right? (Not to say I can judge how well Christensen characterized Fisher, of course.) $\endgroup$
    – Richard Hardy
    Jan 9, 2019 at 18:18
  • 1
    $\begingroup$ Are you referring to p. 124, "[the hypothesis should be rejected] if any relevant feature of the observational record can be shown to [be] sufficiently rare"? That does make it sound like Fisher would have preferred density to the tail proportions. I don't really know for sure. It was just my hunch. $\endgroup$
    – gung - Reinstate Monica
    Jan 9, 2019 at 19:20
  • 1
    $\begingroup$ I see. 2 paragraphs back (p. 124) he quotes Fisher, "In choosing the grounds upon which a general hypothesis should be rejected, personal judgement may and should properly be exercised.” But then turns right around & seems to contradict him, "Nevertheless, the logic of Fisherian testing in no way depends on the source of the test statistic." Your 3rd quote follows from him favoring his interpretation of what the logic of Fisherian testing implies over what Fisher himself appears to have had in mind. $\endgroup$
    – gung - Reinstate Monica
    Jan 9, 2019 at 21:06
  • 1
    $\begingroup$ I don't really have a dog in that fight, but my hunch, above, seems to go with what Fisher says in his quote, whereas the notion that you must use density as opposed to tail proportion goes with what Christensen believes Fisher was obligated to advocate. $\endgroup$
    – gung - Reinstate Monica
    Jan 9, 2019 at 21:08

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.