In a comment to Jaap's answer to this question about reporting tiny p-values, amoeba says "Once the $p$-value is below 0.0001 or something ... it's probably more meaningful to look at $z$-value than at $p$-value," for example the 5$\sigma$ convention in particle physics. There was agreement that this was right. Why is it right? Why are $z$-values a more meaningful way to describe data that gives very low $p$-values? What does "more meaningful" mean?

Quoting Glen_b's answer to why very low $p$-values aren't that meaningful:

But statistical meaning will have been lost far earlier. Note that $p$-values depend on assumptions, and the further out into the extreme tail you go the more heavily the true $p$-value (rather than the nominal value we calculate) will be affected by the mistaken assumptions, in some cases even when they're only a little bit wrong. Since the assumptions are simply not going to be all exactly satisfied, middling $p$-values may be reasonably accurate (in terms of relative accuracy, perhaps only out by a modest fraction), but extremely tiny $p$-values may be out by many orders of magnitude.

Why doesn't similar logic apply to really large $z$-values? Don't they have assumptions that can be violated?

Related questions: If this is the case, why not just always use $z$-values? Are $p$-values "more meaningful" than $z$-values when the $p$-value is high?

  • 5
    $z$-value is just an estimate of some parameter divided by its standard error. No assumptions are needed to compute it, assumptions would be needed to interpret it. The p-value provides such an interpretation. When we say that $z=1.96$ corresponds to $p=.05$ we are making the normality assumption. If we obtain $z=100$ then as Glen_b explains, it does not make sense to convert it into the exact p-value anymore. However, one might want to convey the message that the p-value would be freaking small. Reporting $z=100$ seems like a good way to convey just that. That's what I meant. – amoeba Jun 28 '17 at 22:57
  • @amoeba, why not develop that into an official answer? – gung Jun 29 '17 at 18:08
  • @amoeba Thanks. I agree about writing it as an official answer. From what I can tell you answered my first question. Also a few more sentences would be really helpful, e.g. my last two questions. – R Greg Stacey Jun 29 '17 at 19:12
up vote 3 down vote accepted

A lower $p$-value does not indicate the assumptions were violated. Some really low $p$-values like $2.22\times10^{-16}$ just indicate the limit of the machine, something called machine epsilon. Once a number gets that low, the machine can't go any lower, so spits it out or zero.

The reason one could report the $z$ at that point is $z$ can be interpreted meaningfully within certain contexts. A small $p$ is just really really really small. Additionally, at that point, it is irrelevant. There is a meaningful difference between $p_a=.35$ and $p_b=.09$. However, there is little substantive difference between $p_1=.0000000001$ and $p_2=.0000000000000000001$, even though $\frac{p_1}{p_2}$ is many times larger than $\frac{p_a}{p_b}$.

Finally, the author claims that as $p$-values get really small, failure to meet statistical assumptions have more effects on the magnitude of the $p$-value. We almost never fully meet assumptions of statistical tests, but if we have a $p$-value of .04, and we did a decent job of meeting the assumptions of the test, its magnitude could vary between .02 and say, .06. However, if we have a $p$-value like $p_2$, the author is saying it could be as different as $p_1$ in reality. So at that level, the number you get is not reliable - it is a false precision.

This does not apply to the $z$ much as the change in $p$, as large as from $p_1$ to $p_2$, is just a 3 point change in $z$ from about 6 to 9. And on $z$, if we are concerned with the uncertainty about it, we can always report its confidence interval. Of course, we could do the same for a tiny $p$ but that would be very unconventional and barely interpretable.

  • I quoted Glen_b's answer for why very low $p$-values aren't that meaningful. – R Greg Stacey Jun 28 '17 at 22:32
  • I'll modify my answer. – Heteroskedastic Jim Jun 28 '17 at 22:34
  • Yeah, thanks. I'm running on fumes. – Heteroskedastic Jim Jun 28 '17 at 23:03
  • Are you just saying that z is more human readable for extreme values? I disagree that "a small p is just really really small" and "there is little substantive difference" between p1 and p2. They're difficult to read, but isn't there a "substantive difference" between 10^-9 and 10^-19? – R Greg Stacey Jun 29 '17 at 19:07
  • Partly that z is human interpretable, but also because you are in error territory, false precision. I will guess that Glen_b means that we know from simulation that as you reduce the nominal alpha level, violations of assumptions matter a lot more. So at .05, maybe not so much but at 10^-19, they matter too much for such p-values to be meaningful in any way. They are really low, as to their true magnitude? We cannot be certain. – Heteroskedastic Jim Jun 29 '17 at 19:46

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.