Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I'm trying to find the p-value from an exact paired permutation test. For small samples it works fine:

> library(exactRankTests)
> n <- 1000
> perm.test(runif(n, 0.9, 1),
            runif(n, 0.8, 1),
            paired=TRUE, exact=TRUE)$p.value
[1] 2.907099e-116

but for bigger samples it gives exactly 0:

> n <- 1500
> perm.test(runif(n, 0.9, 1),
            runif(n, 0.8, 1),
            paired=TRUE, exact=TRUE)$p.value
[1] 0

How can I obtain the exact value? And if I cannot, how should I report the p-value in a journal article?

share|improve this question
    
The help on perm.test says that the first two arguments are expected to be integers, and describes a somewhat surprising set of steps when they aren't ... and even then, it looks like it assumes the smallest gap in values is less than tol (is it?). It then links to a description of how the p-values are computed (in the help of another function in the same package). People attempting to answer might need to be aware of this. If anyone is inclined to look at code, it's in perm.test.default in the abovementioned package. –  Glen_b Jul 31 at 23:38

2 Answers 2

You can just say that the p-value is approximately 0 or smaller than $10^{-100}$. For practical purposes the exact value of such a small p-value is irrelevant and might even be unreliable.

Take a look at the following discussion by Andrew Gelman on this topic:

http://andrewgelman.com/2013/01/25/extreem-p-values/

There is also a similar discussion in another question:

Why does R have a minimum p-value of $\le$ 2.22e-16? How should p be reported in such cases?

share|improve this answer

You are not actually getting a p-value of 0 - you're getting p-value that's been rounded to 0 because, as @Karri suggests from this question, it's smaller than the smallest double precision floating point number.

You need not report an exact p-value, indeed very small p-values, or p-values reported with large numbers of significant digits, are something of a bad practice, as they imply a level of precision in your data that isn't actually there. The easiest way is to pick something that most people would read as very significant, and report it as less than that number. For example, p < 0.001. There's not that much information to be had by reporting a smaller, more precise number.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.