# High variance of the distribution of p-values (an argument in Taleb 2016)

I'm trying to understand the big picture claim made in Taleb, 2016, The Meta-Distribution of Standard P-Values.

In it, Taleb makes the following argument for the unreliability of the p-value (as I understand it):

An estimation procedure operating on $n$ data points coming from some distribution $X$ outputs a p value. If we draw n more points from this distribution and output another p value, we can average these p-values obtaining in the limit the so-called "true p-value".

This "true p-value" is shown to have a disturbingly high variance, so that a distribution+procedure with "true p value" $.12$ will 60% of the time report a p-value of <.05.

Question: how can this be reconciled with the traditional argument in favor of the $p$-value. As I understand it, the p-value is supposed to tell you what percentage of the time your procedure will give you the correct interval (or whatever). However, this paper seems to argue that this interpretation is misleading since the p-value will not be the same if you run the procedure again.

Am I missing the point?

• Can you explain what this "traditional argument" is? I am not sure I am clear what argument you're considering. – Glen_b Jun 28 '16 at 2:39
• The question is interesting and is related to a literature for which CV even has a tag, combining-p-values which you might like to add if you think it appropriate. – mdewey Jun 28 '16 at 7:55
• I believe the question about reproducibility of p-values may be very closely related to this one. Perhaps the analysis there is similar to (or even the same) as the one mentioned here. – whuber Jun 28 '16 at 12:52

A p-value is a random variable.

Under $H_0$ (at least for a continuously-distributed statistic), the p-value should have a uniform distribution

For a consistent test, under $H_1$ the p-value should go to 0 in the limit as sample sizes increase toward infinity. Similarly, as effect sizes increase the distributions of p-values should also tend shift toward 0, but it will always be "spread out".

The notion of a "true" p-value sounds like nonsense to me. What would it mean, either under $H_0$ or $H_1$? You might for example say that you mean "the mean of the distribution of p-values at some given effect size and sample size", but then in what sense do you have convergence where the spread should shrink? It's not like you can increase sample size while you hold it constant.

Here's an example with one sample t-tests and a small effect size under $H_1$. The p-values are nearly uniform when the sample size is small, and the distribution slowly concentrates toward 0 as sample size increases. This is exactly how p-values are supposed to behave - for a false null, as the sample size increases, the p-values should become more concentrated at low values, but there's nothing to suggest that the distribution of the values it takes when you make a type II error - when the p-value is above whatever your significance level is - should somehow end up "close" to that significance level.

What, then, would a p-value be an estimate of? It's not like it's converging to something (other than to 0). It's not at all clear why one would expect a p-value to have low variance anywhere but as it approaches 0, even when the power is quite good (e.g. for $\alpha=0.05$, power in the n=1000 case there is close to 57%, but it's still perfectly possible to get a p-value way up near 1)

It's often helpful to consider what's happening both with the distribution of whatever test statistic you use under the alternative and what applying the cdf under the null as a transformation to that will do to the distribution (that will give the distribution of the p-value under the specific alternative). When you think in these terms it's often not hard to see why the behavior is as it is.

The issue as I see it is not so much that there's any inherent problem with p-values or hypothesis testing at all, it's more a case of whether the hypothesis test is a good tool for your particular problem or whether something else would be more appropriate in any particular case -- that's not a situation for broad-brush polemics but one of careful consideration of the kind of questions that hypothesis tests address and the particular needs of your circumstance. Unfortunately careful consideration of these issues are rarely made -- all too often one sees a question of the form "what test do I use for these data?" without any consideration of what the question of interest might be, let alone whether some hypothesis test is a good way to address it.

One difficulty is that hypothesis tests are both widely misunderstood and widely misused; people very often think they tell us things that they don't. The p-value is possibly the single most misunderstood thing about hypothesis tests.

• I think the convergence of the $p$-value is defined with fixed $n$, but $m$ replications of the experiment. Unless I have missed something. – Lepidopterist Jun 30 '16 at 15:09
• @Lepidopterist Replications at fixed $n$ would just be sampling from the distribution of p-values at that $n$. At given $n$, the p-value is a random variable; I show distributions of samples from some examples above. What you converge to is not some "true" p-value but the smooth population versions of the sorts of distributions I show above. – Glen_b Jun 30 '16 at 15:41
• But if you have a random variable, you can talk about it's expectation. On average, the p-value under H1 (under a certain model) may be .12. I guess Taleb's criticism seems strange to me. He seems to be saying that under $H_1$ this expectation may be .12 but may often be less than .05, but this seems to be fine since $H_1$ is in fact true even if it's expectatoin is > .05 – Lepidopterist Jun 30 '16 at 16:07
• I don't think we disagree presently but just to be clear about the situation -- there's no sense in which the mean of the distribution of the p-value at some alternative at some given sample size is the "true" p-value, any more than "3.5" is the true outcome of a die roll. There's no sense in which the p-value converges to that mean value -- when you take a sample of size $n$ from your population, you just get a single p-value drawn from its distribution. – Glen_b Jun 30 '16 at 16:33
• +1. One related -- and fun -- analysis that comes to my mind is what Geoff Cumming calls "A dance of p-values": see youtube.com/watch?v=5OL1RqHrZQ8 (the "dance" happens at around 9 minutes mark). This whole little presentation basically emphasizes how variable the p-values are even for relatively high power. I don't quite agree with Cumming's main point that confidence intervals are so much better than p-values (and I hate that he calls it "new statistics"), but I do think that this amount variability is surprising for many people and the "dance" is a cute way to demonstrate it. – amoeba Dec 6 '16 at 23:31

Glen_b's answer is spot on (+1; consider mine supplemental). The paper you reference by Taleb is topically very similar to a series of papers within the psychology and statistics literature about what kind of information you can glean from analyzing distributions of p-values (what the authors call p-curve; see their site with a bunch of resources, including a p-curve analysis app here).

The authors propose two primary uses of p-curve:

1. You can appraise the evidential value of a literature by analyzing the literature's p-curve. This was their first advertised use of p-curve. Essentially, as Glen_b describes, when you're dealing with non-zero effect sizes, you should see p-curves that are positively skewed below the conventional threshold of p < .05, as smaller p-values should be more likely than p-values closer to p = .05 when an effect (or group of effects) are "real". You can therefore test a p-curve for significant positive skew as a test of evidentiary value. Conversely, the developers propose that you can perform a test of negative skew (i.e., more borderline significant p-valuesthan smaller ones) as a way to test if a given set of effects has been subject to various questionable analytic practices.
2. You can calculate a publication-bias free meta-analytic estimate of effect size using p-curve with published p-values. This one is a bit trickier to explain succinctly, and instead, I'd recommend that you check out their effect-size-estimation focused papers (Simonsohn, Nelson, & Simmons, 2014a, 2014b) and read up on the methods yourself. But essentially, the authors suggest that p-curve can be used to skirt the issue of the file-drawer effect, when conducting a meta-analysis.