60
$\begingroup$

For some tests in R, there is a lower limit on the calculations of $2.22 \cdot 10^{-16}$. I'm not sure why it's this number, if there is a good reason for it or if it's just arbitrary. A lot of other stats packages just go to 0.0001, so this is a much higher level of precision. But I haven't seen too many papers reporting $p < 2.22\cdot 10^{-16}$ or $p = 2.22\cdot 10^{-16}$.

Is it a common/best practice to report this computed value or is it more typical to report something else (like p < 0.000000000000001)?

$\endgroup$
  • $\begingroup$ If you get such small p-value and want to calculate the actual p-value, you can use this function in excel =TDIST(t, df, 2) Add the values of your 't' and df and you will get the actual p-value ta $\endgroup$ – user133545 Oct 5 '16 at 8:40
  • 7
    $\begingroup$ @Tahzeeb is there any reason why Excel would return more precise estimate then R..? As far as I know, it is much less precise. $\endgroup$ – Tim Oct 5 '16 at 10:31
  • $\begingroup$ ...But I haven't seen too many papers reporting p<2.22⋅10−16.... See some GWAS papers, there are many papers showing results for pvalues in hundreds, e.g.: Prostate cancer KLK region, p = 9x10^-186. $\endgroup$ – zx8754 Feb 10 '18 at 21:21
  • 1
    $\begingroup$ See also whuber's answer here: stats.stackexchange.com/questions/11812. $\endgroup$ – amoeba May 9 '18 at 19:56
85
$\begingroup$

There's a good reason for it.

The value can be found via noquote(unlist(format(.Machine)))

           double.eps        double.neg.eps           double.xmin 
         2.220446e-16          1.110223e-16         2.225074e-308 
          double.xmax           double.base         double.digits 
        1.797693e+308                     2                    53 
      double.rounding          double.guard     double.ulp.digits 
                    5                     0                   -52 
double.neg.ulp.digits       double.exponent        double.min.exp 
                  -53                    11                 -1022 
       double.max.exp           integer.max           sizeof.long 
                 1024            2147483647                     4 
      sizeof.longlong     sizeof.longdouble        sizeof.pointer 
                    8                    12                     4 

If you look at the help, (?".Machine"):

double.eps  

the smallest positive floating-point number x such that 1 + x != 1. It equals 
double.base ^ ulp.digits if either double.base is 2 or double.rounding is 0; 
otherwise, it is (double.base ^ double.ulp.digits) / 2. Normally 2.220446e-16.

It's essentially a value below which you can be quite confident the value will be pretty numerically meaningless - in that any smaller value isn't likely to be an accurate calculation of the value we were attempting to compute. (Having studied a little numerical analysis, depending on what computations were performed by the specific procedure, there's a good chance numerical meaninglessness comes in a fair way above that.)

But statistical meaning will have been lost far earlier. Note that p-values depend on assumptions, and the further out into the extreme tail you go the more heavily the true p-value (rather than the nominal value we calculate) will be affected by the mistaken assumptions, in some cases even when they're only a little bit wrong. Since the assumptions are simply not going to be all exactly satisfied, middling p-values may be reasonably accurate (in terms of relative accuracy, perhaps only out by a modest fraction), but extremely tiny p-values may be out by many orders of magnitude.

Which is to say that usual practice (something like the "<0.0001" that's you say is common in packages, or the APA rule that Jaap mentions in his answer) is probably not so far from sensible practice, but the approximate point at which things lose meaning beyond saying 'it's very very small' will of course vary quite a lot depending on circumstances.

This is one reason why I can't suggest a general rule - there can't be a single rule that's even remotely suitable for everyone in all circumstances - change the circumstances a little and the broad grey line marking the change from somewhat meaningful to relatively meaningless will change, sometimes by a long way.

If you were to specify sufficient information about the exact circumstances (e.g. it's a regression, with this much nonlinearity, that amount of variation in this independent variable, this kind and amount of dependence in the error term, that kind of and amount of heteroskedasticity, this shape of error distribution), I could simulate 'true' p-values for you to compare with the nominal p-values, so you could see when they were too different for the nominal value to carry any meaning.

But that leads us to the second reason why - even if you specified enough information to simulate the true p-values - I still couldn't responsibly state a cut-off for even those circumstances.

What you report depends on people's preferences - yours, and your audience. Imagine you told me enough about the circumstances for me to decide that I wanted to draw the line at a nominal $p$ of $10^{-6}$.

All well and good, we might think - except your own preference function (what looks right to you, were you to look at the difference between nominal p-values given by stats packages and the the ones resulting from simulation when you suppose a particular set of failures of assumptions) might put it at $10^{-5}$ and the editors of the journal you want to submit to might put have their blanket rule to cut off at $10^{-4}$, while the next journal might put it at $10^{-3}$ and the next may have no general rule and the specific editor you got might accept even lower values than I gave ... but one of the referees may then have a specific cut off!

In the absence of knowledge of their preference functions and rules, and the absence of knowledge of your own utilities, how do I responsibly suggest any general choice of what actions to take?

I can at least tell you the sorts of things that I do (and I don't suggest this is a good choice for you at all):

There are few circumstances (outside of simulating p-values) in which I would make much of a p less than $10^{-6}$ (I may or may not mention the value reported by the package, but I wouldn't make anything of it other than it was very small, I would usually emphasize the meaningless of the exact number). Sometimes I take a value somewhere in the region of $10^{-5}$ to $10^{-4}$ and say that p was much less than that. On occasion I do actually do as suggested above - perform some simulations to see how sensitive the p-value is in the far tail to various violations of the assumptions, particularly if there's a specific kind of violation I am worried about.

That's certainly helpful in informing a choice - but I am as likely to discuss the results of the simulation as to use them to choose a cut-off-value, giving others a chance to choose their own.

An alternative to simulation is to look at some procedures that are more robust* to the various potential failures of assumption and see how much difference to the p-value that might make. Their p-values will also not be particularly meaningful, but they do at least give some sense of how much impact there might be. If some are very different from the nominal one, it also gives more of an idea which violations of assumptions to investigate the impact of. Even if you don't report any of those alternatives, it gives a better picture of how meaningful your small p-value is.

* Note that here we don't really need procedures that are robust to gross violations of some assumption; ones that are less affected by relatively mild deviations of the relevant assumption should be fine for this exercise.

I will say that when/if you do come to do such simulations, even with quite mild violations, in some cases it can be surprising at how far even not-that-small p-values can be wrong. That has done more to change the way I personally interpret a p-value more than it has shifted the specific cut-offs I might use.

When submitting the results of an actual hypothesis test to a journal, I try to find out if they have any rule. If they don't, I tend to please myself, and then wait for the referees to complain.

$\endgroup$
  • 11
    $\begingroup$ I especially like the comment on statistical meaning being lost far earlier. $\endgroup$ – usεr11852 Dec 7 '13 at 14:21
  • $\begingroup$ Great answer! I appreciate all the detail on this, it clears up why R gives this number. But it doesn't really answer the question of what to report. $\endgroup$ – paul Dec 7 '13 at 19:58
  • 1
    $\begingroup$ I rather felt I had addressed the issue, in the sense that I explained why it wasn't responsible to make a specific suggestion. Note that I do discuss why it makes sense to report something like the "<0.0001" that's common practice in some packages. There's a couple of reasons why I don't suggest a specific number - the first of which I gave. I will expand on that reason and the second one in an edit. $\endgroup$ – Glen_b Dec 7 '13 at 22:40
  • $\begingroup$ paul, I have added some more substantial discussion. $\endgroup$ – Glen_b Dec 7 '13 at 23:04
  • 2
    $\begingroup$ Yes, you do need to do something; the point of my more extensive commentary was to convey that I can't tell you what you should choose to do, I can only discuss the issues that come into your choice. I hope I have done so, but I am happy to try to clarify any issues further if I can. $\endgroup$ – Glen_b Dec 8 '13 at 5:28
27
$\begingroup$

What common practice is might depend on your field of research. The manual of the American Psychological Association (APA), which is one of the most often used citation styles, states (p. 139, 6th edition):

Do not use any value smaller than p < 0.001

$\endgroup$
  • 8
    $\begingroup$ Although this is what I also usually cite (+1), I am not sure whether or not one needs to revise this recommendation by one decimal place, given the recent recommendation of Valen Johnson in PNAS: "Make 0.005 the default level of significance [...]. Associate highly significant test results with P values that are less than 0.001." $\endgroup$ – Henrik Dec 7 '13 at 14:28
  • 3
    $\begingroup$ Good answer. There's no style guides and no real standards in my fields, at least not for p-values. I do interdisciplinary work but I guess computer science and HCI would be the field for this. I think APA style would be where authors would turn, since the methods are generally borrowed from cognitive psych or other areas that the APA would cover. $\endgroup$ – paul Dec 7 '13 at 20:04
  • 10
    $\begingroup$ Particle physics uses a $5\sigma$ rule (you might have seen it in the news with the confirmation of the Higgs boson), which blows way past this limit (being smaller even than $p < 10^{-6}$). Standards differ by area! $\endgroup$ – Glen_b May 20 '14 at 4:51
  • 1
    $\begingroup$ @Glen_b: Good point about $5\sigma$ in particle physics, but I guess what you wrote in your answer about sensitivity to assumptions etc. explains (or is at least part of the reason) why they report sigmas (i.e. basically $z$-statistic) instead of $p$-values. Once the $p$-value is below $0.0001$ or something (my usual advice is to report as many zeroes as one feels comfortable to print without switching to exponential notation), it's probably more meaningful to look at $z$-value than at $p$-value. $\endgroup$ – amoeba Feb 12 '17 at 10:00
  • $\begingroup$ @amoeba Yeah, I think you're right. $\endgroup$ – Glen_b Feb 12 '17 at 10:10
14
$\begingroup$

Such extreme p-values occur more often in fields with very large amounts of data, such as genomics and process monitoring. In those cases, it's sometimes reported as -log10(p-value). See for example, this figure from Nature, where the p-values go down to 1e-26.

-log10(p-value) is called "LogWorth" by statisticians I work with at JMP.

$\endgroup$
  • 21
    $\begingroup$ This is true, and worth pointing out, but it may also be worth mentioning that in this case the $p$-value should be really thought of only as an index of signal strength -- such small $p$-values (sometimes even if corrected for multiple comparisons) are so tiny that the probability that the NSA broke in and tampered with your data (and then brainwashed you so you can't remember) is far, far, higher than the nominal $p$-value. $\endgroup$ – Ben Bolker Jan 22 '14 at 20:51
  • 8
    $\begingroup$ @BenBolker Indeed, while less probable than "the NSA tampered with your data", even events like "A cosmic ray flipped several important bits in your data" are far, far more likely than those probabilities. $\endgroup$ – Glen_b May 20 '14 at 3:41
  • 5
    $\begingroup$ In a 2015 neuroscience paper published in Nature, authors report $p<10^{-100}$ a couple of times when presenting correlation coefficients ($\rho\approx0.9$ and $n\sim 500$). Made me smile and remember your comments here, @Ben and Glen_b. $\endgroup$ – amoeba Apr 16 '15 at 15:43
  • 8
    $\begingroup$ Here is a new finding in my quest for the minimal p-value reported in the literature: another 2015 neuroscience paper published in Nature (from a group that has just got 2014 Nobel prize, by the way) reports $p=2.2\times 10^{-226}$. Wow. (The paper is actually still great.) Cc to @Glen_b. $\endgroup$ – amoeba Jul 9 '15 at 16:05
  • 8
    $\begingroup$ @amoeba Over in the Slate Star Codex comment section, Daniel Wells notes that science.sciencemag.org/content/363/6425/eaau1043 reports a p-value of 3.6e-2382 ("not a typo, two thousand", says Daniel), which beats yours by quite a margin! $\endgroup$ – Mark Amery May 20 at 10:15
-3
$\begingroup$

in R, "<2e-16" doesn't mean literally <2e-16, but instead, it means the value is so small that R can not record or display it.

In regression test, I often get p as small as 4.940656e-324, when it outputs "<2e-16", this is a number even smaller than 4.940656e-324

$\endgroup$
  • $\begingroup$ Which number is "even smaller than 4.940656e-324"? $\endgroup$ – Sven Hohenstein Aug 18 '17 at 5:26
  • 8
    $\begingroup$ Your statement "in R, "<2e-16" doesn't mean literally <2e-16" is incorrect. When R displays <2e-16, the value is smaller than 2e-16, literally. $\endgroup$ – Sven Hohenstein Aug 18 '17 at 5:29
  • $\begingroup$ You mis-understood what I said. When R said "<2e-16", the p-value is smaller than 2e-16, but it doesn't mean p-value smaller than 2e-16 will be displayed as "<2e-16". As I have shown, R has no problem to display in lm summary p-value of any number between [4.940656e-324, 2e-16], while the left boundary is 2^-1074. So I assume, only when the p-value is smaller than 2^-1074, R will then say the p-value is smaller than a small delta value. It just happens that R display this delta value as 2e-16. So my guess is "<2e-16" actually means "<2^-1074" in p-values $\endgroup$ – user3590816 Aug 22 '17 at 4:57
  • 6
    $\begingroup$ Your guess, however, is incorrect: that's what @Sven is trying to tell you. See the help for format.pval or simply try it out, as in format.pval(1e-16). $\endgroup$ – whuber Aug 31 '17 at 19:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.