32
$\begingroup$

I had a discussion with a statistician back in 2009 where he stated that the exact value of a p-value is irrelevant: the only thing that is important is whether it is significant or not. I.e. one result cannot be more significant than another; your samples for example, either come from the same population or don't.

I have some qualms with this, but I can perhaps understand the ideology:

  1. The 5% threshold is arbitrary, i.e. that p = 0.051 is not significant and that p = 0.049 is, shouldn't really change the conclusion of your observation or experiment, despite one result being significant and the other not significant.

    The reason I bring this up now is that I'm studying for an MSc in Bioinformatics, and after talking to people in the field, there seems to be a determined drive to get an exact p-value for every set of statistics they do. For instance, if they 'achieve' a p-value of p < 1.9×10-12, they want to demonstrate HOW significant their result is, and that this result is SUPER informative. This issue exemplified with questions such as: Why can't I get a p-value smaller than 2.2e-16?, whereby they want to record a value that indicates that by chance alone this would be MUCH less than 1 in a trillion. But I see little difference in demonstrating that this result would occur less than 1 in a trillion as opposed to 1 in a billion.

  2. I can appreciate then that p < 0.01 shows that there is less than 1% chance that this would occur, whereas p < 0.001 indicates that a result like this is even more unlikely than the aforementioned p-value, but should your conclusions drawn be completely different? After all they are both significant p-values. The only way I can conceive of wanting to record the exact p-value is during a Bonferroni correction whereby the threshold changes due to the number of comparisons made, thus decreasing the type I error. But even still, why would you want to show a p-value that is 12 orders of magnitude smaller than your threshold significance?

  3. And isn't applying the Bonferroni correction in itself slightly arbitrary too? In the sense that initially the correction is seen as very conservative, and therefore there are other corrections that one can choose to access the significance level that the observer could use for their multiple comparisons. But because of this, isn't the point at which something becomes significant essentially variable depending upon what statistics the researcher wants to use. Should statistics be so open to interpretation?

In conclusion, shouldn't statistics be less subjective (although I guess the need for it to be subjective is as a consequence of a multivariate system), but ultimately I want some clarification: can something be more significant than something else? And will p < 0.001 suffice in respect to trying to record the exact p-value?

$\endgroup$
5
  • 7
    $\begingroup$ This is quite interesting: stat.washington.edu/peter/342/nuzzo.pdf $\endgroup$
    – Dan
    Apr 24, 2014 at 6:43
  • 4
    $\begingroup$ Loosely related: In my answer to the question When to use Fisher and Neyman-Pearson framework, I argue that there is a role for each framework. In keeping w/ my position there, I would say that exact p-values wouldn't matter in w/i the N-P framework, but can w/i the Fisherian framework (to the extent that number of digits reported are actually reliable). $\endgroup$ Apr 24, 2014 at 18:30
  • $\begingroup$ It is amazing how much some statisticians want to hold on to the concept of a p-value when it is usually the right answer to the wrong question. Suppose p-values weren't implemented in any stats software package. I doubt that people would write their own code to get it. $\endgroup$ May 7, 2014 at 12:24
  • 3
    $\begingroup$ @probabilityislogic - having cut my statistical teeth on permutation tests, p-values are a very natural way to think in that case, so I might just write my own code to get them if they weren't ... and in fact, on the very rare occasions when I do tests at all, they're usually for some atypical situation requiring simulation or some form of resampling, I've found I actually tend do so. I'd tend to say instead that hypothesis tests usually answer the wrong question. On the rare occasion that they do, I think they have value (not least, other people aren't bound by my significance level). $\endgroup$
    – Glen_b
    May 10, 2014 at 8:02
  • $\begingroup$ @glen_b - my problem with p-values is that the don't provide "the answer" to any hypothesis test on their own, as they ignore alternatives. If you are restricted to just one number, then the value of the likelihood for the data is a much better statistic than the p-value (as well as having the same problems as p). This way people aren't bound by your choice of test statistic (in addition to not being bound by your threshold for significance). $\endgroup$ May 10, 2014 at 11:12

3 Answers 3

25
+50
$\begingroup$
  1. The type 1 / false rejection error rate $\alpha=.05$ isn't completely arbitrary, but yes, it is close. It's somewhat preferable to $\alpha=.051$ because it's less cognitively complex (people like round numbers and multiples of five). It's a decent compromise between skepticism and practicality, though maybe a little outdated – modern methods and research resources may make higher standards (i.e., lower $p$ values) preferable, if standards there must be (Johnson, 2013).

    IMO, the greater problem than the choice of threshold is the often unexamined choice to use a threshold where it is not necessary or helpful. In situations where a practical choice has to be made, I can see the value, but much basic research does not necessitate the decision to dismiss one's evidence and give up on the prospect of rejecting the null just because a given sample's evidence against it falls short of almost any reasonable threshold. Yet much of this research's authors feel obligated to do so by convention, and resist it uncomfortably, inventing terms like "marginal" significance to beg for attention when they can feel it slipping away because their audiences often don't care about $p$s $\ge.05$. If you look around at other questions here on $p$ value interpretation, you'll see plenty of dissension about the interpretation of $p$ values by binary fail to/reject decisions regarding the null.

  2. Completely different – no. Meaningfully different – maybe. One reason to show a ridiculously small $p$ value is to imply information about effect size. Of course, just reporting effect size would be much better for several technical reasons, but authors often fail to consider this alternative, and audiences may be less familiar with it as well, unfortunately. In a null-hypothetical world where no one knows how to report effect sizes, one may be right most often in guessing that a smaller $p$ means a larger effect. To whatever extent this null-hypothetical world is closer to reality than the opposite, maybe there's some value in reporting exact $p$s for this reason. Please understand that this point is pure devil's advocacy...

    Another use for exact $p$s that I've learned by engaging in a very similar debate here is as indices of likelihood functions. See Michael Lew's comments on and article (Lew, 2013) linked in my answer to "Accommodating entrenched views of p-values".

  3. I don't think the Bonferroni correction is the same kind of arbitrary really. It corrects the threshold that I think we agree is at least close-to-completely arbitrary, so it doesn't lose any of that fundamental arbitrariness, but I don't think it adds anything arbitrary to the equation. The correction is defined in a logical, pragmatic way, and minor variations toward larger or smaller corrections would seem to require rather sophisticated arguments to justify them as more than arbitrary, whereas I think it would be easier to argue for an adjustment of $\alpha$ without having to overcome any deeply appealing yet simple logic in it.

    If anything, I think $p$ values should be more open to interpretation! I.e., whether the null is really more useful than the alternative ought to depend on more than just the evidence against it, including the cost of obtaining more information and the added incremental value of more precise knowledge thusly gained. This is essentially the Fisherian no-threshold idea that, AFAIK, is how it all began. See "Regarding p-values, why 1% and 5%? Why not 6% or 10%?"

If fail to/reject crises aren't forced upon the null hypothesis from the outset, then the more continuous understanding of statistical significance certainly does admit the possibility of continuously increasing significance. In the dichotomized approach to statistical significance (I think this is sometimes referred to as the Neyman-Pearson framework; cf. Dienes, 2007), no, any significant result is as significant as the next – no more, no less. This question may help explain that principle: "Why are p-values uniformly distributed under the null hypothesis?" As for how many zeroes are meaningful and worth reporting, I recommend Glen_b's answer to this question: "How should tiny $p$-values be reported? (and why does R put a minimum on 2.22e-16?)" – it's much better than the answers to the version of that question you linked on Stack Overflow!

References
- Johnson, V. E. (2013). Revised standards for statistical evidence. Proceedings of the National Academy of Sciences, 110(48), 19313–19317. Retrieved from http://www.pnas.org/content/110/48/19313.full.pdf.
- Lew, M. J. (2013). To P or not to P: On the evidential nature of P-values and their place in scientific inference. arXiv:1311.0081 [stat.ME]. Retrieved from http://arxiv.org/abs/1311.0081.

$\endgroup$
6
  • 4
    $\begingroup$ +1, a lot of good thoughts here. 1 quibble though, re #1, I would say we should often have lower standards (i.e., higher p-values) as preferable. It is often difficult to get enough data to have good power to study something. I've run a number of power analyses for doctors who want to study a rare condition. They say, 'this is really understudied, I have an idea for a new approach, we can probably get 50 patients w/ this over the next two years', & I say 'your power will be 45%', and the project is abandoned. Rare diseases will continue to be understudied if p must be .05 or less. $\endgroup$ Apr 24, 2014 at 18:19
  • 2
    $\begingroup$ @gung: I agree completely. I cited Johnson (2013) only because I was aware of his argument, not because I agree with it :) IMO, having one conventional standard that is inflexible and insensitive to the concerns you describe (which echo my point in the second paragraph of my response to #3) is one of the core problems, and adjusting it up or down isn't going to solve it. When there is no real need for a hard-and-fast fail to/reject decision, I think it's much better to make the judgment of how valuable one's evidence is based on much more than the probability of the sample given the null. $\endgroup$ Apr 24, 2014 at 18:24
  • 5
    $\begingroup$ Excellent discussion. An interesting article of some relevance is Gelman and Stern's The difference between “significant” and “non-significant” is not itself statistically significant (later published in American Statistician, 2006), which I wouldn't say characterizes the value of p as necessarily meaningless but would inject a strong note of caution in regard to putting much emphasis on comparing p-values (rather than effect estimates, say). Gelman has discussed issues related to this frequently on his blog. $\endgroup$
    – Glen_b
    May 6, 2014 at 23:08
  • 2
    $\begingroup$ For 2 I think it should be emphasized that $p$ values should NOT be used as measures of association or effect. A desirable property of an inferential test is consistency, that is as sample size goes to infinity, the power of the test goes to 1, or $p$ values go to 0. So $p$ values should not be used to describe the effect/association. $\endgroup$
    – bdeonovic
    May 7, 2014 at 1:50
  • 2
    $\begingroup$ It seems Gelman provides a link to the pdf of the published paper on his site also. $\endgroup$
    – Glen_b
    May 7, 2014 at 2:28
13
$\begingroup$

It seems to me that, if a value is meaningful, its exact value is meaningful.

The p value answers this question:

If, in the population from which this sample was randomly drawn, the null hypothesis was true, what is the probability of getting a test statistic at least as extreme as the one we got in the sample?

What about this definition makes an exact value meaningless?

This is a different question from the ones about extreme values of p. The problem with statements that involve p with many 0's are about how well we can estimate p in the extremes. Since we can't do that very well, it makes no sense to use such precise estimates of p. This is the same reason we don't say that p = 0.0319281010012981 . We don't know those last digits with any confidence.

Should our conclusions be different if p < 0.001 rather than p < 0.05? Or, to use precise numbers, should our conclusions be different if p = 0.00023 rather than p = 0.035?

I think the problem is with how we typically conclude things about p. We say "significant" or "not significant" based on some arbitrary level. If we use these arbitrary levels, then, yes, our conclusions will be different. But this is not how we should be thinking about these things. We should be looking at the weight of evidence and statistical tests are only part of that evidence. I will (once again) plug Robert Abelson's "MAGIC criteria":

Magnitude - how big is the effect?

Articulation - how precisely is it stated? Are there lots of exceptions?

Generality - to what group does it apply?

Interestingness - will people care?

Credibility - does it make sense?

It is the combination of all of these that matters. Note that Abelson doesn't mention p values at all, although they do come in as a sort of hybrid of magnitude and articulation.

$\endgroup$
5
  • 6
    $\begingroup$ We don't often say it, but technically the p-value is only reflecting something about the "probability of getting a test statistic at least as extreme as the one we got in the sample" if the null hypothesis is true, our sample estimate of the population variance is perfectly accurate, and we meet all of the other assumptions of our test. Throw some confidence intervals around some p-values via bootstrapping and I think you'll see that frequently we aren't all that confident about the hundredths place either. $\endgroup$ Apr 30, 2014 at 20:47
  • 3
    $\begingroup$ In short, it is such a convoluted counter-factual that attempting to quantify a p-value is counter productive when we really should (as you imply) get back to the MAGIC. $\endgroup$ Apr 30, 2014 at 20:48
  • $\begingroup$ I have to admit, I hadn't thought of putting confidence intervals (or credibility intervals) around p values. I wonder how much has been done in this area? $\endgroup$
    – Peter Flom
    Apr 30, 2014 at 20:56
  • 3
    $\begingroup$ I don't have a citation handy, but I know there is work along those lines - regardless, it is an academic thing to do because you can make confidence intervals of your confidence intervals of your confidence intervals nearly ad infinitum (there is a maximum variance that is reasonably estimated from any set of data). I had a rather long and detailed conversation along these lines with @Nick Stauner once upon a time. He may still have some the articles he dug up during that conversation to bring to the table. $\endgroup$ May 1, 2014 at 1:40
  • 1
    $\begingroup$ Nothing on confidence intervals for p values that I recall, but I might've skimmed over those sections. I wasn't interested in making confidence intervals for p values either ;) $\endgroup$ May 6, 2014 at 22:34
0
$\begingroup$

Suppose you collect data and compute a $p$-value without first selecting a Type I error rate. Then, observing the $p$-value, you select a rate specifically so that it will be statistically significant. If you then proclaim, "My result is statistically significant!" that statement alone contains no inferentially-useful information. What makes that statement informative, in the general case, is having chosen the error rate in advance$-$or, equivalently, the pre-existence of a default error rate of 5%. For purposes of the null hypothesis statistical test (NHST), one does not reject at $p = .049$ and "super-reject" at $p = .01$.

That said, there are other uses for the $p$-value beyond NHST's, which may use the exact observed value. For example, the $p$-value of a $t$-test is a fit statistic for the model where population $t$ has a non-central $t$-distribution. As with $R^2$, one may compare different $p$-values for different models applied to the same data, e.g., for $t$-tests with different experimental hypotheses of different non-centrality parameters. This makes $p$ equivalent to a test of the effect size, as another answer to your question implies. (However, this is rarely the interpretation intended for tables of multi-starred estimates.)

$\endgroup$
5
  • $\begingroup$ You were going great up until you stated "Rather, it means we would observe it 1% of the time +/- some (typically large) margin of error." I'm having trouble finding an interpretation of the setting that makes that statement correct. As a point of departure, bear in mind (1) the p-value is hypothetical: it is computed supposing the null is true and (2) the p-value is usually exactly and correctly computed: it is what it is, for your dataset. $\endgroup$
    – whuber
    Nov 3, 2023 at 18:48
  • $\begingroup$ Since p is an exact function of the statistic theta-hat, it must have a sampling distribution exactly correlated with that of theta-hat. If the reality is that theta = 0, p has a null distribution with an expected value and a standard error. Yes, once you have a sample in hand, p is what it is, but theta-hat also is what it is. One can nonetheless construct a CI around either, no? In other words, if you say you'll reject the null 5% of the time when theta = 0, you're implying a population distribution of p-values, from which you will draw a p-value <= .05 5% of the time. $\endgroup$
    – virtuolie
    Nov 3, 2023 at 21:15
  • $\begingroup$ A CI obtains its meaning relative to a parameter. (Part of its definition concerns the chance that the CI covers the parameter's value.) What parameter is the p-value estimating?? $\endgroup$
    – whuber
    Nov 3, 2023 at 22:01
  • $\begingroup$ Actually, Neyman says: "The parameter is an unknown constant, and no probability statement concerning its value may be made." Nevertheless, whether a CI for p is valid or useful is beside the point: I was only underlining that p varies over samples. Observing p=.05 doesn't guarantee this sample is randomly drawn 5% of the time from the true null population. A sample with an unrepresentative SE, for example, gives an inaccurate p-value. But an experiment with a nominal .05 alpha guarantees one rejects 5% of the time. p depends on the data, the decision rule does not. $\endgroup$
    – virtuolie
    Nov 3, 2023 at 23:51
  • 1
    $\begingroup$ Mea culpa! Yes, of course, if the assumptions are correct and the population is t-distributed (say), then the p-value is independent of the sample because the sd cancels in the computed t. Thank you for your patience with my brain fart! $\endgroup$
    – virtuolie
    Nov 4, 2023 at 1:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.