# What is a good, convincing example in which p-values are useful?

My question in the title is self explanatory, but I would like to give it some context.

The ASA released a statement earlier this week “on p-values: context, process, and purpose”, outlining various common misconceptions of the p-value, and urging caution in not using it without context and thought (which could be said just about any statistical method, really).

In response to the ASA, professor Matloff wrote a blog post titled: After 150 Years, the ASA Says No to p-values. Then professor Benjamini (and I) wrote a response post titled It’s not the p-values’ fault – reflections on the recent ASA statement. In response to it professor Matloff asked in a followup post:

What I would like to see [... is] — a good, convincing example in which p-values are useful. That really has to be the bottom line.

To quote his two major arguments against the usefulness of the $p$-value:

1. With large samples, significance tests pounce on tiny, unimportant departures from the null hypothesis.

2. Almost no null hypotheses are true in the real world, so performing a significance test on them is absurd and bizarre.

I am very interested in what other crossvalidated community members think of this question/arguments, and of what may constitute a good response to it.

• Notice another two threads related to this topic: stats.stackexchange.com/questions/200500/… and stats.stackexchange.com/questions/200745/… – Tim Mar 11 '16 at 11:56
• Thanks Tim. I suspect my question is different enough that it deserves its own thread (especially since it was not answered in the two you mentioned). Still, the links are very interesting! – Tal Galili Mar 11 '16 at 12:10
• It deserves and is interesting (hence my +1), I provided the links just FYI :) – Tim Mar 11 '16 at 12:16
• I must say that I have not (yet) read what Matloff wrote on the topic, but still, in order for your question to stand on its own, can you perhaps briefly summarize why he finds any standard example of p-values usage not "good/convincing"? E.g. somebody wants to study if a certain experimental manipulation changes animal behavior in a particular direction; so an experimental and a control groups are measured and compared. As a reader of such a paper, I am happy to see the p-value (i.e. they are useful for me), because if it's large then I don't need to pay attention. This example is not enough? – amoeba Mar 11 '16 at 13:14
• @amoeba - he lists them here: matloff.wordpress.com/2016/03/07/… ----- Quoting his arguments: 1) with large samples, significance tests pounce on tiny, unimportant departures from the null hypothesis. 2) Almost no null hypotheses are true in the real world, so performing a significance test on them is absurd and bizarre. ----- I have my own take on these (which I would like to later formalize), but I am sure others will have insightful ways of answering this. – Tal Galili Mar 11 '16 at 13:35

I will consider both Matloff's points:

1. With large samples, significance tests pounce on tiny, unimportant departures from the null hypothesis.

The logic here is that if somebody reports highly significant $p=0.0001$, then from this number alone we cannot say if the effect is large and important or irrelevantly tiny (as can happen with large $n$). I find this argument strange and cannot connect to it at all, because I have never seen a study that would report a $p$-value without reporting [some equivalent of] effect size. Studies that I read would e.g. say (and usually show on a figure) that group A had such and such mean, group B had such and such mean and they were significantly different with such and such $p$-value. I can obviously judge for myself if the difference between A and B is large or small.

(In the comments, @RobinEkman pointed me to several highly-cited studies by Ziliak & McCloskey (1996, 2004) who observed that the majority of the economics papers trumpet "statistical significance" of some effects without paying much attention to the effect size and its "practical significance" (which, Z&MS argue, can often be minuscule). This is clearly bad practice. However, as @MatteoS explained below, the effect sizes (regression estimates) are always reported, so my argument stands.)

2. Almost no null hypotheses are true in the real world, so performing a significance test on them is absurd and bizarre.

This concern is also often voiced, but here again I cannot really connect to it. It is important to realize that researchers do not increase their $n$ ad infinitum. In the branch of neuroscience that I am familiar with, people will do experiments with $n=20$ or maybe $n=50$, say, rats. If there is no effect to be seen then the conclusion is that the effect is not large enough to be interesting. Nobody I know would go on breeding, training, recording, and sacrificing $n=5000$ rats to show that there is some statistically significant but tiny effect. And whereas it might be true that almost no real effects are exactly zero, it is certainly true that many many real effects are small enough to be detected with reasonable sample sizes that reasonable researchers are actually using, exercising their good judgment.

(There is a valid concern that sample sizes are often not big enough and that many studies are underpowered. So perhaps researchers in many fields should rather aim at, say, $n=100$ instead of $n=20$. Still, whatever the sample size is, it puts a limit on the effect size that the study has power to detect.)

In addition, I do not think I agree that almost no null hypotheses are true, at least not in the experimental randomized studies (as opposed to observational ones). Two reasons:

• Very often there is a directionality to the prediction that is being tested; researcher aims to demonstrate that some effect is positive $\delta>0$. By convention this is usually done with a two-sided test assuming a point null $H_0: \delta=0$ but in fact this is rather a one-sided test trying to reject $H_0: \delta<0$. (@CliffAB's answer, +1, makes a related point.) And this can certainly be true.

• Even talking about the point "nil" null $H_0: \delta=0$, I do not see why they are never true. Some things are just not causally related to other things. Look at the psychology studies that are failing to replicate in the last years: people feeling the future; women dressing in red when ovulating; priming with old-age-related words affecting walking speed; etc. It might very well be that there are no causal links here at all and so the true effects are exactly zero.

Himself, Norm Matloff suggests to use confidence intervals instead of $p$-values because they show the effect size. Confidence intervals are good, but notice one disadvantage of a confidence interval as compared to the $p$-value: confidence interval is reported for one particular coverage value, e.g. $95\%$. Seeing a $95\%$ confidence interval does not tell me how broad a $99\%$ confidence interval would be. But one single $p$-value can be compared with any $\alpha$ and different readers can have different alphas in mind.

In other words, I think that for somebody who likes to use confidence intervals, a $p$-value is a useful and meaningful additional statistic to report.

I would like to give a long quote about the practical usefulness of $p$-values from my favorite blogger Scott Alexander; he is not a statistician (he is a psychiatrist) but has lots of experience with reading psychological/medical literature and scrutinizing the statistics therein. The quote is from his blog post on the fake chocolate study which I highly recommend. Emphasis mine.

[...] But suppose we're not allowed to do $p$-values. All I do is tell you "Yeah, there was a study with fifteen people that found chocolate helped with insulin resistance" and you laugh in my face. Effect size is supposed to help with that. But suppose I tell you "There was a study with fifteen people that found chocolate helped with insulin resistance. The effect size was $0.6$." I don't have any intuition at all for whether or not that's consistent with random noise. Do you? Okay, then they say we’re supposed to report confidence intervals. The effect size was $0.6$, with $95\%$ confidence interval of $[0.2, 1.0]$. Okay. So I check the lower bound of the confidence interval, I see it’s different from zero. But now I’m not transcending the $p$-value. I’m just using the p-value by doing a sort of kludgy calculation of it myself – “$95\%$ confidence interval does not include zero” is the same as “$p$-value is less than $0.05$”.

(Imagine that, although I know the $95\%$ confidence interval doesn’t include zero, I start wondering if the $99\%$ confidence interval does. If only there were some statistic that would give me this information!)

But wouldn’t getting rid of $p$-values prevent “$p$-hacking”? Maybe, but it would just give way to “d-hacking”. You don’t think you could test for twenty different metabolic parameters and only report the one with the highest effect size? The only difference would be that p-hacking is completely transparent – if you do twenty tests and report a $p$ of $0.05$, I know you’re an idiot – but d-hacking would be inscrutable. If you do twenty tests and report that one of them got a $d = 0.6$, is that impressive? [...]

But wouldn’t switching from $p$-values to effect sizes prevent people from making a big deal about tiny effects that are nevertheless statistically significant? Yes, but sometimes we want to make a big deal about tiny effects that are nevertheless statistically significant! Suppose that Coca-Cola is testing a new product additive, and finds in large epidemiological studies that it causes one extra death per hundred thousand people per year. That’s an effect size of approximately zero, but it might still be statistically significant. And since about a billion people worldwide drink Coke each year, that’s a ten thousand deaths. If Coke said “Nope, effect size too small, not worth thinking about”, they would kill almost two milli-Hitlers worth of people.

For some further discussion of various alternatives to $p$-values (including Bayesian ones), see my answer in ASA discusses limitations of $p$-values - what are the alternatives?

• Your response to the second argument misses the point, in my opinion. No one is suggesting that real researchers increase their sample sizes ad infinitum. The point (as I see it) is that any null hypothesis of the form "effect = 0" that a researcher would be interested in testing is going to be false, and there's little value in performing a hypothesis test if the null hypothesis is already known to be false. This of course assumes that what we are really interested in is the relevant population parameter(s), rather than the characteristics of the sample. – mark999 Mar 11 '16 at 21:53
• But I admit that "any null hypothesis ... is going to be false" is only an assumption. – mark999 Mar 11 '16 at 21:56
• I should admit that my reasoning here was rather informal and I never tried to formalize it. Perhaps to make this argument work, I should not say that there is a clear boundary between interesting and uninteresting effect sizes. Rather it is a continuum with interestingness increasing further away from zero, and the "reasonable" sample size should give small power to the very uninteresting effect sizes and large power to the very interesting ones, but there is no one threshold. I wonder if one can accurately formalize it along the Neyman-Pearson lines. – amoeba Mar 11 '16 at 23:22
• Maybe you "have never seen a study that would report a $p$-value without reporting [some equivalent of] effect size", but Ziliak and McCloskey found some 300 such papers published in just one journal, The American Economic Review, during just two decades. Such papers made up more than 70% of all the papers they looked at. – Robin Ekman Mar 12 '16 at 15:24
• @amoeba: the source of the 70% claim may be the ambiguous phrasing in the 2006 abstract: “of the 182 full-length papers published in the 1980s in the [AER] 70% did not distinguish economic from statistical significance”. What they mean by this–as explained in both papers–is that often only the latter is commented upon, and that the magnitude of the regression coefficient in relation to the dependent variable (“economic significance” in their jargon) is not as extensively analyzed. But it is always reported. I suggest you edit your update in the answer to reflect that :-) – MatteoS Mar 13 '16 at 17:50

I take great offense at the following two ideas:

1. With large samples, significance tests pounce on tiny, unimportant departures from the null hypothesis.

2. Almost no null hypotheses are true in the real world, so performing a significance test on them is absurd and bizarre.

It is such a strawman argument about p-values. The very foundational problem that motivated the development of statistics comes from seeing a trend and wanting to know whether what we see is by chance, or representative of a systematic trend.

With that in mind, it is true that we, as statisticians, do not typically believe that a null-hypothesis is true (i.e. $$H_o: \mu_d = 0$$, where $$\mu_d$$ is the mean difference in some measurement between two groups). However, with two sided tests, we don't know which alternative hypothesis is true! In a two sided test, we may be willing to say that we are 100% sure that $$\mu_d \neq 0$$ before seeing the data. But we do not know whether $$\mu_d > 0$$ or $$\mu_d < 0$$. So if we run our experiment and conclude that $$\mu_d > 0$$, we have rejected $$\mu_d = 0$$ (as Matloff might say; useless conclusion) but more importantly, we have also rejected $$\mu_d < 0$$ (I say; useful conclusion). As @amoeba pointed out, this also applies to one sided test that have the potential to be two sided, such as testing whether a drug has a positive effect.

It's true that this doesn't tell you the magnitude of the effect. But it does tell you the direction of the effect. So let's not put the cart before the horse; before I start drawing conclusions about the magnitude of the effect, I want to be confident I've got the direction of the effect correct!

Similarly, the argument that "p-values pounce on tiny, unimportant effects" seems quite flawed to me. If you think of a p-value as a measure of how much the data supports the direction of your conclusion, then of course you want it to pick up small effects when the sample size is large enough. To say this means they are not useful is very strange to me: are these fields of research that have suffered from p-values the same ones that have so much data they have no need to assess the reliability of their estimates? Similarly, if your issues is really that p-values "pounce on tiny effect sizes", then you can simply test the hypotheses $$H_{1}:\mu_d > 1$$ and $$H_{2}: \mu_d < -1$$ (assuming you believe 1 to be the minimal important effect size). This is done often in clinical trials.

To further illustrate this, suppose we just looked at confidence intervals and discarded p-values. What is the first thing you would check in the confidence interval? Whether the effect was strictly positive (or negative) before taking the results too seriously. As such, even without p-values, we would informally be doing hypothesis testing.

Finally, in regards to the OP/Matloff's request, "Give a convincing argument of p-values being significantly better", I think question is a little awkward. I say this because depending on your view, it automatically answers itself ("give me one concrete example where testing a hypothesis is better than not testing them"). However, a special case that I think is almost undeniable is that of RNAseq data. In this case, we are typically looking at the expression level of RNA in two different groups (i.e. diseased, controls) and trying to find genes that are differentially expressed in the two groups. In this case, the effect size itself is not even really meaningful. This is because the expression levels of different genes vary so wildly that for some genes, having 2x higher expression doesn't mean anything, while on other tightly regulated genes, 1.2x higher expression is fatal. So the actual magnitude of the effect size is actually somewhat uninteresting when first comparing the groups. But you really, really want to know if the expression of the gene changes between the groups and direction of the change! Furthermore, it's much more difficult to address the issues of multiple comparisons (for which you may be doing 20,000 of them in a single run) with confidence intervals than it is with p-values.

• I disagree that knowing the direction of the effect is in itself useful. If I spit on the ground, I know this will either improve or inhibit plant growth (i.e. the null hypothesis of no effect is false). How is knowing the direction of this effect without any information on its magnitude helpful? Yet this is the only thing the p-value from your two-sided test / two one-sided tests (sort of) tells you! (BTW, I think the ‘spit on the ground’ example was borrowed from some paper on p-values I read years ago, but I can’t recall which one.) – Karl Ove Hufthammer Mar 12 '16 at 17:48
• @KarlOveHufthammer: Cart before the horse. I shouldn't stop just because I know the direction of the effect. But I should care that I have the direction correct before I start worrying about the magnitude. Do you think the scientific community would be better off by embracing everything with large estimated effects without checking p-values? – Cliff AB Mar 12 '16 at 18:27
• Furthermore, this idea that "p-values don't give you useful information" is just sloppy use of hypothesis testing. You can easily test the hypotheses of $H_a: \mu_d > 1$ and $H_a: \mu_d < -1$ if you think an effect size must be of magnitude greater than 1 to be in anyway meaningful. (edited the answer to reflect this, as I believe it is an important point. Thanks for bringing it up) – Cliff AB Mar 12 '16 at 18:38
• You made several very good points in the edits. I really like your answer now! – amoeba Mar 12 '16 at 21:34
• While working on my answer to stats.stackexchange.com/questions/200500 I came across this recent preprint by Wagenmakers et al where they essentially argue your point about directionality: "one-sided P values can be given a Bayesian interpretation as an approximate test of direction, that is, a test of whether a latent effect is negative or positive." It's interesting because Wagenmakers is a die-hard Bayesian, he wrote a lot against p-values. Still, I see some conceptual agreement here. – amoeba Mar 15 '16 at 11:59

Forgive my sarcasm, but one obvious good example of the utility of p-values is in getting published. I had one experimenter approach me for producing a p-value... he had introduced a transgene in a single plant to improve growth. From that single plant he produced multiple clones and chose the largest clone, an example where the entire population is enumerated. His question, the reviewer wants to see a p-value that this clone is the largest. I mentioned that there is not any need for statistics in this case as he had the entire population at hand, but to no avail.

More seriously, in my humble opinion, from an academic perspective i find these discussion interesting and stimulating, just like the frequentist vs Bayesian debates from a few years ago. It brings out the differing perspectives of the best minds in this field and illuminates the many assumptions/pitfalls associated with the methodology thats not generally readily accesible.

In practice, I think that rather than arguing about the best approach and replacing one flawed yardstick with another, as has been suggested before elsewhere, for me it is rather a revelation of an underlying systemic problem and the focus should be on trying to find optimal solutions. For instance, one could present situations where p-values and CI complement each other and circumstance wherein one is more reliable than the other. In the grand scheme of things, I understand that all inferential tools have their own shortcomings which need to be understood in any application so as to not stymie progress towards the ultimate goal.. the deeper understanding of the system of study.

I'll give you the exemplary case of how p-values should be used and reported. It's a very recent report on the search of a mysterious particle on Large Hadron Collider(LHC) in CERN.

A few months ago there was a lot of excited chatter in high energy physics circles about a possibility that a large particle was detected on LHC. Remember this was after Higgs boson discovery. Here's the excerpt from the paper "Search for resonances decaying to photon pairs in 3.2 fb−1 of p p collisions at √s = 13 TeV with the ATLAS detector" by The ATLAS Collaboration Dec 15 2015 and my comments follow:

What they're saying here is that the event counts exceed what the Standard Model predicts. The Figure below from the paper shows the p-values of excess events as a function of a mass of a particle. You see how p-value dives around 750 GeV. So, they're saying that there's a possibility that a new particle is detected with a mass equal to 750 Giga eV. The p-values on the figure are calculated as "local". The global p-values are much higher. That's not important for our conversation though.

What's important is that p-values are not yet "low enough" for physicists to declare a find, but "low enough" to get excited. So, they're planning to keep counting, and hoping that that p-values will further decrease.

Zoom a few months forward to Aug 2016, Chicago, a conference on HEP. There was a new report presented "Search for resonant production of high mass photon pairs using 12.9 fb−1 of proton-proton collisions at √ s = 13 TeV and combined interpretation of searches at 8 and 13 TeV" by The CMS Collaboration this time. Here's the excerpts with my comments again:

So, the guys continued collecting events, and now that blip of excess events at 750 GeV is gone. The figure below from the paper shows p-values, and you can see how p-value increased compared to the first report. So, they sadly conclude that no particle is detected at 750 GeV.

I think this is how p-values are supposed to be used. They totally make a sense, and they clearly work. I think the reason is that frequentist approaches are inherently natural in physics. There's nothing subjective about particle scattering. You collect a a sample large enough and you get a clear signal if it's there.

If you're really into how exactly p-values are calculated here, read this paper: "Asymptotic formulae for likelihood-based tests of new physics" by Cowan et al

• Everybody was hoping that the 750 GeV peak is real and is now sad. But I was actually hoping it would turn out to be a fluctuation (and could have bet it would) and am now relieved. I think it's cool that standard model works so well. Don't quite understand the burning desire to move beyond standard model (as if everything else in physics is solved). Anyway, +1, good example. – amoeba Aug 6 '16 at 23:51

The other explanations are all fine, I just wanted to try and give a brief and direct answer to the question that popped into my head.

Checking Covariate Imbalance in Randomized Experiments

Your second claim (about unrealistic null hypotheses) is not true when we are checking covariate balance in randomized experiments where we know the randomization was done properly. In this case, we know that the null hypothesis is true. If we get a significant difference between treatment and control group on some covariate - after controlling for multiple comparisons, of course - then that tells us that we got a "bad draw" in the randomization and we maybe shouldn't trust the causal estimate as much. This is because we might think that our treatment effect estimates from this particular "bad draw" randomization are further away from the true treatment effects than estimates obtained from a "good draw."

I think this is a perfect use of p-values. It uses the definition of p-value: the probability of getting a value as or more extreme given the null hypothesis. If the result is highly unlikely, then we did in fact get a "bad draw."

Balance tables/statistics are also common when using observational data to try and make causal inferences (e.g., matching, natural experiments). Although in these cases balance tables are far from sufficient to justify a "causal" label to the estimates.

• I disagree that this is a perfect (or even good) use of p-values. How do you define a "bad draw"? – mark999 Mar 12 '16 at 6:28
• @mark, Okay. I think I can reply your last question while Matt is away: of course in the sample. Imagine a randomized experiment with 50 people. Imagine that it just so happened that all 25 people in group A turned out to be men and all 25 people in group B turned out to be women. It's pretty obvious that this can cast serious doubts on any conclusions of the study; that's an example of a "bad draw". Matt suggested to run a test for differences in gender (covariate) between A and B. I don't see how Matt's answer can be interpreted differently. There are arguably no populations here at all. – amoeba Mar 12 '16 at 22:34
• @mark999 But a test for difference between 12/25 and 13/25 will obviously yield high non-significant p-value, so I am not sure what is your point here. Matt suggested to run a test and consider a low p-value as a red flag. No red flag in your example. I think I will stop here and let Matt continue the dialog if he wants. – amoeba Mar 12 '16 at 23:26
• No. See 'balance test fallacy': gking.harvard.edu/files/matchse.pdf You describe a case where the test statistic itself may be fine (used as a distance measure to minimise) but a p-value for it makes no sense. – conjugateprior Mar 13 '16 at 22:14
• For a more recent examination of this in psycho- and neurolinguistics, there is a new arXiv preprint. When you're deliberating manipulating balance, etc., you're not random sampling and even if you were, the tests answer a different inferential question about balance in population not balance in the sample. – Livius Jun 10 '16 at 3:33

Error rates control is similar to quality control in production. A robot in a production line has a rule for deciding that a part is defective which guarantees not to exceed a specified rate of defective parts that go through undetected. Similarly, an agency that makes decisions for drug approval based on "honest" P-values has a way to keep the rate of false rejections at a controlled level, by definition via the frequentist long-run construction of tests. Here, "honest" means absence of uncontrolled biases, hidden selections, etc.

However, neither the robot, nor the agency have a personal stake in any particular drug or a part that goes through the assembly conveyor. In science, on the other hand, we, as individual investigators care most about the particular hypothesis we study, rather than about the proportion of spurious claims in our favorite journal we submit to. Neither the P-value magnitude nor the bounds of a confidence interval (CI) refer directly to our question about the credibility of what we report. When we construct the CI bounds, we should be saying that the only meaning of the two numbers is that if other scientists do the same kind of CI computation in their studies, the 95% or whatever coverage will be maintained over various studies as a whole.

In this light, I find it ironic that P-values are being "banned" by journals, considering that in the thick of replicability crisis they are of more value to journal editors than to researchers submitting their papers, as a practical way of keeping the rate of spurious findings reported by a journal at bay, in the long run. P-values are good at filtering, or as IJ Good wrote, they are good for protecting statistician's rear end, but not so much the rear end of the client.

P.S. I'm a huge fan of Benjamini and Hochberg's idea of taking the unconditional expectation across studies with multiple tests. Under the global "null", the "frequentist" FDR is still controlled - studies with one or more rejections pop up in a journal at a controlled rate, although, in this case, any study where some rejections have been actually made has the proportion of false rejections that is equal to one.

I agree with Matt that p-values are useful when the null hypothesis is true.

The simplest example I can think of is testing a random number generator. If the generator is working correctly, you can use any appropriate sample size of realizations and when testing the fit over many samples, the p-values should have a uniform distribution. If they do, this is good evidence for a correct implementation. If they don't, you know you have made an error somewhere.

Other similar situations occur when you know a statistic or random variable should have a certain distribution (again, the most obvious context is simulation). If the p-values are uniform, you have found support for a valid implementation. If not, you know you have a problem somewhere in your code.

I can think of example in which p-values are useful, in Experimental High Energy Physics. See Fig.1 This plot is taken from this paper: Observation of a new particle in the search for the Standard Model Higgs boson with the ATLAS detector at the LHC

In this Fig, the p-value is shown versus the mass of an hypothetical particle. The null hypothesis denotes the compatibility of the observation with a continuous background. The large ($5 \sigma$) deviation at m$_\mathrm{H} \approx 125$ GeV was the first evidence and discovery of a new particle. This earned François Englert, Peter Higgs the Nobel Prize in Physics in 2013.

• You need to provide more information about the plot, with background and how it addresses the original question. This is not nearly enough information. – Greenparker Jun 7 '16 at 17:08
• @Greenparker, tried to add some background on the plot. – Nicolas Gutierrez Jun 7 '16 at 18:27
• You didn't explain what is a blue band $\pm 1\sigma$ – Aksakal Aug 7 '16 at 15:36