3
$\begingroup$

I am trying to see if I understand the definition of $p$-value as used by Sir R. A. Fisher and the one used today by frequentist statisticians (not sure how to call it better).

$p$-value according to Sir R. A. Fisher
A $p$-value for a given realization of a test statistic is the probability of getting an equally extreme or a more extreme realization under repeated sampling, provided that the null hypothesis $H_0$ is true. Extremity is defined in terms of the density of the test statistic (a random variable) given $H_0$, with "more extreme" meaning "having lower density". Note that such a definition of $p$-value does not involve any reference to an alternative hypothesis. Based on my rudimentary knowledge of history of statistics, I suppose Fisher would be fine with such a definition. As expressed by Christensen (2005),

The $p$-value is the probability of seeing something as weird or weirder than you actually saw. (There is no need to specify that it is computed under the null hypothesis because there is only one hypothesis.)

Also there:

The key fact in a Fisherian test is that it makes no reference to any alternative hypothesis.

Also (square brackets added by me as deemphasis):

[First, $F$ tests and $\chi^2$ tests are typically rejected only for large values of the test statistic. Clearly, in Fisherian testing, that is inappropriate.] Finding the $p$ value for an $F$ test should involve finding the density associated with the observed $F$ statistic and finding the probability of getting any value with a lower density. [This will be a two-tailed test, rejecting for values that are very large or very close to 0.]

$p$-value used by frequentist statisticians today
I think the definition above (i.e. my attempt at rephrasing Fisher's definition) is not what is normally used today, because every now and then I encounter "one-sided" vs. "two-sided" $p$-values where extremity is clearly defined with reference to a specific alternative hypothesis. The low-density areas on the "uninteresting" side of $H_0$ are thus not counted. Otherwise the definition is the same.

Questions

  1. Does my understanding of $p$-value according to Sir R. A. Fisher make sense?
    (Or did I overlook something important or made some mistakes in my explanation?)
  2. Is my understanding of today's definition of the $p$-value correct?

References

|cite|improve this question
$\endgroup$
  • 2
    $\begingroup$ I don't know if he would have used density per se. Imagine a test statistic whose distribution was bi-modal under the null (I can't think of one, but nothing prevents a test static from having a weird distribution). In that case, a "low density" could occur in the middle of the distribution, but my intuition is that Fisher wouldn't have the same interpretation there as in the tails. $\endgroup$ – gung♦ Jan 9 at 18:03
  • $\begingroup$ @gung, well, that is exactly why density is useful - it allows us to deal with ugly distributions. If all densities were concave, we could go for critical values and intervals of points futher away from the centre than critical values, skipping references to density. This is not to say that Fisher should share my sentiment, of course, but your point goes against the third quote of Christensen (2005), right? (Not to say I can judge how well Christensen characterized Fisher, of course.) $\endgroup$ – Richard Hardy Jan 9 at 18:18
  • $\begingroup$ Are you referring to p. 124, "[the hypothesis should be rejected] if any relevant feature of the observational record can be shown to [be] sufficiently rare"? That does make it sound like Fisher would have preferred density to the tail proportions. I don't really know for sure. It was just my hunch. $\endgroup$ – gung♦ Jan 9 at 19:20
  • $\begingroup$ @gung, this is also relevant, but I was mainly referring to the third quote. Thanks for your comments! $\endgroup$ – Richard Hardy Jan 9 at 19:53
  • $\begingroup$ What "third quote"? From a skim of the article, I didn't easily see what you meant. $\endgroup$ – gung♦ Jan 9 at 20:09

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.