7
$\begingroup$

I am writing the results of my thesis, and I have a few questions on how to report P-values.

  1. When is it appropriate to give the exact P-value, instead of writing e.g. P<0.05 (also in case of non-significant P-values)?
  2. If I have a P-value of e.g. 0.016, should I report it as P=0.01 or P=0.02? Or if it was 0.0167, should I write 0.016 or 0.017?
  3. When I have a very small P-value, e.g. 0.00000013, is it appropriate to report it as P<0.000001, or is it better to stop at e.g. P<0.001?
$\endgroup$

2 Answers 2

7
$\begingroup$

When is it appropriate to give the exact P-value, instead of writing e.g. P<0.05 (also in case of non-significant P-values)?

As a general guideline you want to convey as much information about your results as possible. When you report a p-value as p=0.016 instead of as p<0.02 then the readers will have a more precise view of the results. Often it is also helpful to report the statistic along with it. For instance you can encounter sentences like: "There was a positive effect of X with a coefficient $\beta_X = 0.5$, which was significant ($t = 2.6$, $p = 0.017$)".

The reporting of only $p<0.05$ can be done when one is only interested in strict cut-off levels (but often the cut-off levels are arbitrary). It is also common in graphs or tables where significance is denoted with superscripts like $^\star: p < 0.05$, ${^\star}{^\star}: p < 0.01$, ${^\star}{^\star}{^\star}: p < 0.001$ and in that case it is used as abbreviation to prevent cluttered graphs and tables that become difficult to read.

Instead of p-values, it is also increasingly more popular to report confidence intervals instead. Then the sentence above would become "There was a significant positive effect of X with a coefficient $\beta_X = 0.5$ (95% confidence interval [0.099,0.901] )".

If I have a P-value of e.g. 0.016, should I report it as P=0.01 or P=0.02?

Rounding off p-values might give a false idea. However, you could round up and use an inequality sign like $P<0.02$.

When I have a very small P-value, e.g. 0.00000013, is it appropriate to report it as P<0.000001, or is it better to stop at e.g. P<0.001?

The precision that is appropriate will depend on the application.

Often more precision than below $0.001$ is not needed. Also a better precision can be deceptive because the computation depends on several assumptions and the p-value is an estimate that can be computed with high precision, but does not truly have that high precision because of uncertainty in the assumptions underlying the computation. The computation with some model can be performed with a high precision, but that doesn't mean that the result of that computation should be considered as precise (the potential errors due to a wrong model should be considered as well).

In exact sciences like physics where high precision measurements are possible and a strong theoretical framework is present it is more common to see high precision p-values. For instance in some fields of physics one desires the significance to be below above $5\sigma$ which is equivalent to $p < 0.00000057$.

$\endgroup$
6
  • $\begingroup$ I have yet to see someone try to work out the numerical error of their p-value from their machine epsilon, or perform a grid search of levels of arbitrary precision to obtain a stable calculation out to some number of digits. But these approaches could hypothetically be taken to account for the numerical imprecision of p-values. But it would definitely add to the labour of the analysis. $\endgroup$
    – Galen
    May 13 at 15:46
  • 2
    $\begingroup$ @Galen I am not referring to computational errors. The computation is often very precise (and that is actually the problem). The problem is that filling in numbers/data into a very precise formula, which gives you a very precise number, does not mean that your research was equally precise. For instance, your formula might have been using the assumption that data is normal distributed, and if that assumption is wrong then the p value might be wrong as well. If we report low precision p values then such systematic errors are less important. $\endgroup$ May 13 at 15:49
  • $\begingroup$ I see my interpretation was off, which more careful reading could have prevented. $\endgroup$
    – Galen
    May 13 at 16:04
  • $\begingroup$ Re "does not truly have that high precision". Since a p-value is a likelihood of a result under a model, might a highly precise p-value truly be precise but just for the wrong model? Analogous to the idea of type 3 error, the error might not be with the precision of the p-value but rather with answering the wrong question. Is that what you are considering, or something else? $\endgroup$
    – Galen
    May 13 at 16:12
  • 1
    $\begingroup$ @Galen This very issue has come up in the context of bootstrapping a p-value (or using a permutation test), because the reported p-value has an inherent binomial uncertainty related to the length of the simulation. In effect, such p-values are computed only with two or three significant digits of precision. $\endgroup$
    – whuber
    May 13 at 17:13
1
$\begingroup$
  1. In practice, people report the exact p-value then declare whether the hypothesis test is statistically significant. Most statisticians don't advocate this. if the goal is hypothesis testing then reporting the p-value is moot because it doesn't make the test more significant, on the other hand the exact p-value matters if the result of the analysis doesn't boil down to a decision theoretic approach. So if the audience is not statisticians, they would probably expect you to do it the wrong way.

  2. The significant digits on a p-value matters. First, you have to follow whatever style guide there is for your thesis or publication. If there are no rules, you need to consider the variability of the analysis. In general, it's hard to say how "stable" a p-value is without simulation or bootstrapping, but you'll often find that even with a huge analysis, a p-value is only really stable out to two decimal places - hence reporting 3 decimal places conveys a level of precision that is not present in the analysis. Secondly, if there is a pre-specified "alpha" level, you need to show enough precision for the reader to know whether p < alpha. In other words, p=0.05 requires two decimal places of precision if p = 0.04 or lower, but borderline cases get awkward. Do you say p = 0.049994 or p = 0.05 but it was statistically significant.

  3. Don't cheat the incidental precision of the analysis. Use the prespecified precision following the rules above, and report out to the smallest decimal. Other aspects of the analysis should convey how striking the result is, a "really small p-value" in and of itself means nothing. And never never report p = 0.00.

$\endgroup$
6
  • 4
    $\begingroup$ Re "if the goal is hypothesis testing then reporting the p-value is moot:" not at all! The main point to reporting it is to enable other users to conduct significance tests with their own thresholds rather than having to accept your test sizes. I would be surprised to see any real evidence that "most statisticians don't advocate this." $\endgroup$
    – whuber
    May 13 at 15:19
  • $\begingroup$ Thank you for your answers. I am still a bit confused about to what point a P-value can be rounded off? So referring to my second question $\endgroup$
    – balzy
    May 13 at 15:22
  • 1
    $\begingroup$ @whuber that's a ludicrous thought to me, because you don't choose an alpha level at the point of analysis, you choose it at the point of design and it consequently informs sample size, design, number of tests, and overall feasibility. Nobody chooses the alpha after all that is said and done. $\endgroup$
    – AdamO
    May 13 at 16:15
  • 1
    $\begingroup$ Re "if the goal is hypothesis testing then reporting the p-value is moot": Perhaps some types of meta-analysis can be performed more precisely if p-values are reported precisely. Otherwise reporting whether a test was significant at some $\alpha$ still allows the use of interval arithmetic by first inferring the interval $p \in [0, \alpha)$, which is less precise but not necessarily useless. $\endgroup$
    – Galen
    May 13 at 16:23
  • 4
    $\begingroup$ I am not saying that. I am pointing out that your readers might have chosen a priori a level they view as more appropriate than your 0.05 (or whatever) level. For instance, many might have adopted a recent suggestion to rely only on p-values less than 0.005. Anyone who censors their reporting by writing "p < 0.05" is thereby making their results almost worthless. They also make multiple comparisons corrections nearly impossible to perform, again a disservice to knowledgeable and thoughtful readers. $\endgroup$
    – whuber
    May 13 at 17:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.