What is a good, convincing example in which p-values are useful?

Question

My question in the title is self explanatory, but I would like to give it some context.

The ASA released a statement earlier this week “on p-values: context, process, and purpose”, outlining various common misconceptions of the p-value, and urging caution in not using it without context and thought (which could be said just about any statistical method, really).

In response to the ASA, professor Matloff wrote a blog post titled: After 150 Years, the ASA Says No to p-values. Then professor Benjamini (and I) wrote a response post titled It’s not the p-values’ fault – reflections on the recent ASA statement. In response to it professor Matloff asked in a followup post:

What I would like to see [... is] — a good, convincing example in which p-values are useful. That really has to be the bottom line.

To quote his two major arguments against the usefulness of the $p$-value:

1. With large samples, significance tests pounce on tiny, unimportant departures from the null hypothesis.

2. Almost no null hypotheses are true in the real world, so performing a significance test on them is absurd and bizarre.

I am very interested in what other crossvalidated community members think of this question/arguments, and of what may constitute a good response to it.

Answer 1

I will consider both Matloff's points:

1. With large samples, significance tests pounce on tiny, unimportant departures from the null hypothesis.

   The logic here is that if somebody reports highly significant $p=0.0001$, then from this number alone we cannot say if the effect is large and important or irrelevantly tiny (as can happen with large $n$). I find this argument strange and cannot connect to it at all, because I have never seen a study that would report a $p$-value without reporting [some equivalent of] effect size. Studies that I read would e.g. say (and usually show on a figure) that group A had such and such mean, group B had such and such mean and they were significantly different with such and such $p$-value. I can obviously judge for myself if the difference between A and B is large or small.

   (In the comments, @RobinEkman pointed me to several highly-cited studies by Ziliak & McCloskey (1996, 2004) who observed that the majority of the economics papers trumpet "statistical significance" of some effects without paying much attention to the effect size and its "practical significance" (which, Z&MS argue, can often be minuscule). This is clearly bad practice. However, as @MatteoS explained below, the effect sizes (regression estimates) are always reported, so my argument stands.)

2. Almost no null hypotheses are true in the real world, so performing a significance test on them is absurd and bizarre.

   This concern is also often voiced, but here again I cannot really connect to it. It is important to realize that researchers do not increase their $n$ ad infinitum. In the branch of neuroscience that I am familiar with, people will do experiments with $n=20$ or maybe $n=50$, say, rats. If there is no effect to be seen then the conclusion is that the effect is not large enough to be interesting. Nobody I know would go on breeding, training, recording, and sacrificing $n=5000$ rats to show that there is some statistically significant but tiny effect. And whereas it might be true that almost no real effects are exactly zero, it is certainly true that many many real effects are small enough to be detected with reasonable sample sizes that reasonable researchers are actually using, exercising their good judgment.

   (There is a valid concern that sample sizes are often not big enough and that many studies are underpowered. So perhaps researchers in many fields should rather aim at, say, $n=100$ instead of $n=20$. Still, whatever the sample size is, it puts a limit on the effect size that the study has power to detect.)

   In addition, I do not think I agree that almost no null hypotheses are true, at least not in the experimental randomized studies (as opposed to observational ones). Two reasons:

   • Very often there is a directionality to the prediction that is being tested; researcher aims to demonstrate that some effect is positive $\delta>0$. By convention this is usually done with a two-sided test assuming a point null $H_0: \delta=0$ but in fact this is rather a one-sided test trying to reject $H_0: \delta<0$. (@CliffAB's answer, +1, makes a related point.) And this can certainly be true.

   • Even talking about the point "nil" null $H_0: \delta=0$, I do not see why they are never true. Some things are just not causally related to other things. Look at the psychology studies that are failing to replicate in the last years: people feeling the future; women dressing in red when ovulating; priming with old-age-related words affecting walking speed; etc. It might very well be that there are no causal links here at all and so the true effects are exactly zero.

Himself, Norm Matloff suggests to use confidence intervals instead of $p$-values because they show the effect size. Confidence intervals are good, but notice one disadvantage of a confidence interval as compared to the $p$-value: confidence interval is reported for one particular coverage value, e.g. $95\%$. Seeing a $95\%$ confidence interval does not tell me how broad a $99\%$ confidence interval would be. But one single $p$-value can be compared with any $\alpha$ and different readers can have different alphas in mind.

In other words, I think that for somebody who likes to use confidence intervals, a $p$-value is a useful and meaningful additional statistic to report.

---

I would like to give a long quote about the practical usefulness of $p$-values from my favorite blogger Scott Alexander; he is not a statistician (he is a psychiatrist) but has lots of experience with reading psychological/medical literature and scrutinizing the statistics therein. The quote is from his blog post on the fake chocolate study which I highly recommend. Emphasis mine.

[...] But suppose we're not allowed to do $p$-values. All I do is tell you "Yeah, there was a study with fifteen people that found chocolate helped with insulin resistance" and you laugh in my face. Effect size is supposed to help with that. But suppose I tell you "There was a study with fifteen people that found chocolate helped with insulin resistance. The effect size was $0.6$." I don't have any intuition at all for whether or not that's consistent with random noise. Do you? Okay, then they say we’re supposed to report confidence intervals. The effect size was $0.6$, with $95\%$ confidence interval of $[0.2, 1.0]$. Okay. So I check the lower bound of the confidence interval, I see it’s different from zero. But now I’m not transcending the $p$-value. I’m just using the p-value by doing a sort of kludgy calculation of it myself – “$95\%$ confidence interval does not include zero” is the same as “$p$-value is less than $0.05$”.

(Imagine that, although I know the $95\%$ confidence interval doesn’t include zero, I start wondering if the $99\%$ confidence interval does. If only there were some statistic that would give me this information!)

But wouldn’t getting rid of $p$-values prevent “$p$-hacking”? Maybe, but it would just give way to “d-hacking”. You don’t think you could test for twenty different metabolic parameters and only report the one with the highest effect size? The only difference would be that p-hacking is completely transparent – if you do twenty tests and report a $p$ of $0.05$, I know you’re an idiot – but d-hacking would be inscrutable. If you do twenty tests and report that one of them got a $d = 0.6$, is that impressive? [...]

But wouldn’t switching from $p$-values to effect sizes prevent people from making a big deal about tiny effects that are nevertheless statistically significant? Yes, but sometimes we want to make a big deal about tiny effects that are nevertheless statistically significant! Suppose that Coca-Cola is testing a new product additive, and finds in large epidemiological studies that it causes one extra death per hundred thousand people per year. That’s an effect size of approximately zero, but it might still be statistically significant. And since about a billion people worldwide drink Coke each year, that’s a ten thousand deaths. If Coke said “Nope, effect size too small, not worth thinking about”, they would kill almost two milli-Hitlers worth of people.

---

For some further discussion of various alternatives to $p$-values (including Bayesian ones), see my answer in ASA discusses limitations of $p$-values - what are the alternatives?

Answer 2

I take great offense at the following two ideas:

1. With large samples, significance tests pounce on tiny, unimportant departures from the null hypothesis.

2. Almost no null hypotheses are true in the real world, so performing a significance test on them is absurd and bizarre.

It is such a strawman argument about p-values. The very foundational problem that motivated the development of statistics comes from seeing a trend and wanting to know whether what we see is by chance, or representative of a systematic trend.

With that in mind, it is true that we, as statisticians, do not typically believe that a null-hypothesis is true (i.e. $H_o: \mu_d = 0$, where $\mu_d$ is the mean difference in some measurement between two groups). However, with two sided tests, we don't know which alternative hypothesis is true! In a two sided test, we may be willing to say that we are 100% sure that $\mu_d \neq 0$ before seeing the data. But we do not know whether $\mu_d > 0$ or $\mu_d < 0$. So if we run our experiment and conclude that $\mu_d > 0$, we have rejected $\mu_d = 0$ (as Matloff might say; useless conclusion) but more importantly, we have also rejected $\mu_d < 0$ (I say; useful conclusion). As @amoeba pointed out, this also applies to one sided test that have the potential to be two sided, such as testing whether a drug has a positive effect.

It's true that this doesn't tell you the magnitude of the effect. But it does tell you the direction of the effect. So let's not put the cart before the horse; before I start drawing conclusions about the magnitude of the effect, I want to be confident I've got the direction of the effect correct!

Similarly, the argument that "p-values pounce on tiny, unimportant effects" seems quite flawed to me. If you think of a p-value as a measure of how much the data supports the direction of your conclusion, then of course you want it to pick up small effects when the sample size is large enough. To say this means they are not useful is very strange to me: are these fields of research that have suffered from p-values the same ones that have so much data they have no need to assess the reliability of their estimates? Similarly, if your issues is really that p-values "pounce on tiny effect sizes", then you can simply test the hypotheses $H_{1}:\mu_d > 1$ and $H_{2}: \mu_d < -1$ (assuming you believe 1 to be the minimal important effect size). This is done often in clinical trials.

To further illustrate this, suppose we just looked at confidence intervals and discarded p-values. What is the first thing you would check in the confidence interval? Whether the effect was strictly positive (or negative) before taking the results too seriously. As such, even without p-values, we would informally be doing hypothesis testing.

Finally, in regards to the OP/Matloff's request, "Give a convincing argument of p-values being significantly better", I think question is a little awkward. I say this because depending on your view, it automatically answers itself ("give me one concrete example where testing a hypothesis is better than not testing them"). However, a special case that I think is almost undeniable is that of RNAseq data. In this case, we are typically looking at the expression level of RNA in two different groups (i.e. diseased, controls) and trying to find genes that are differentially expressed in the two groups. In this case, the effect size itself is not even really meaningful. This is because the expression levels of different genes vary so wildly that for some genes, having 2x higher expression doesn't mean anything, while on other tightly regulated genes, 1.2x higher expression is fatal. So the actual magnitude of the effect size is actually somewhat uninteresting when first comparing the groups. But you really, really want to know if the expression of the gene changes between the groups and direction of the change! Furthermore, it's much more difficult to address the issues of multiple comparisons (for which you may be doing 20,000 of them in a single run) with confidence intervals than it is with p-values.

Answer 3

Forgive my sarcasm, but one obvious good example of the utility of p-values is in getting published. I had one experimenter approach me for producing a p-value... he had introduced a transgene in a single plant to improve growth. From that single plant he produced multiple clones and chose the largest clone, an example where the entire population is enumerated. His question, the reviewer wants to see a p-value that this clone is the largest. I mentioned that there is not any need for statistics in this case as he had the entire population at hand, but to no avail.

More seriously, in my humble opinion, from an academic perspective i find these discussion interesting and stimulating, just like the frequentist vs Bayesian debates from a few years ago. It brings out the differing perspectives of the best minds in this field and illuminates the many assumptions/pitfalls associated with the methodology thats not generally readily accesible.

In practice, I think that rather than arguing about the best approach and replacing one flawed yardstick with another, as has been suggested before elsewhere, for me it is rather a revelation of an underlying systemic problem and the focus should be on trying to find optimal solutions. For instance, one could present situations where p-values and CI complement each other and circumstance wherein one is more reliable than the other. In the grand scheme of things, I understand that all inferential tools have their own shortcomings which need to be understood in any application so as to not stymie progress towards the ultimate goal.. the deeper understanding of the system of study.

Answer 4

I'll give you the exemplary case of how p-values should be used and reported. It's a very recent report on the search of a mysterious particle on Large Hadron Collider(LHC) in CERN.

A few months ago there was a lot of excited chatter in high energy physics circles about a possibility that a large particle was detected on LHC. Remember this was after Higgs boson discovery. Here's the excerpt from the paper "Search for resonances decaying to photon pairs in 3.2 fb−1 of p p collisions at √s = 13 TeV with the ATLAS detector" by The ATLAS Collaboration Dec 15 2015 and my comments follow:

What they're saying here is that the event counts exceed what the Standard Model predicts. The Figure below from the paper shows the p-values of excess events as a function of a mass of a particle. You see how p-value dives around 750 GeV. So, they're saying that there's a possibility that a new particle is detected with a mass equal to 750 Giga eV. The p-values on the figure are calculated as "local". The global p-values are much higher. That's not important for our conversation though.

What's important is that p-values are not yet "low enough" for physicists to declare a find, but "low enough" to get excited. So, they're planning to keep counting, and hoping that that p-values will further decrease.

Zoom a few months forward to Aug 2016, Chicago, a conference on HEP. There was a new report presented "Search for resonant production of high mass photon pairs using 12.9 fb−1 of proton-proton collisions at √ s = 13 TeV and combined interpretation of searches at 8 and 13 TeV" by The CMS Collaboration this time. Here's the excerpts with my comments again:

So, the guys continued collecting events, and now that blip of excess events at 750 GeV is gone. The figure below from the paper shows p-values, and you can see how p-value increased compared to the first report. So, they sadly conclude that no particle is detected at 750 GeV.

I think this is how p-values are supposed to be used. They totally make a sense, and they clearly work. I think the reason is that frequentist approaches are inherently natural in physics. There's nothing subjective about particle scattering. You collect a a sample large enough and you get a clear signal if it's there.

If you're really into how exactly p-values are calculated here, read this paper: "Asymptotic formulae for likelihood-based tests of new physics" by Cowan et al

Answer 5

The other explanations are all fine, I just wanted to try and give a brief and direct answer to the question that popped into my head.

Checking Covariate Imbalance in Randomized Experiments

Your second claim (about unrealistic null hypotheses) is not true when we are checking covariate balance in randomized experiments where we know the randomization was done properly. In this case, we know that the null hypothesis is true. If we get a significant difference between treatment and control group on some covariate - after controlling for multiple comparisons, of course - then that tells us that we got a "bad draw" in the randomization and we maybe shouldn't trust the causal estimate as much. This is because we might think that our treatment effect estimates from this particular "bad draw" randomization are further away from the true treatment effects than estimates obtained from a "good draw."

I think this is a perfect use of p-values. It uses the definition of p-value: the probability of getting a value as or more extreme given the null hypothesis. If the result is highly unlikely, then we did in fact get a "bad draw."

Balance tables/statistics are also common when using observational data to try and make causal inferences (e.g., matching, natural experiments). Although in these cases balance tables are far from sufficient to justify a "causal" label to the estimates.

Answer 6

Error rates control is similar to quality control in production. A robot in a production line has a rule for deciding that a part is defective which guarantees not to exceed a specified rate of defective parts that go through undetected. Similarly, an agency that makes decisions for drug approval based on "honest" P-values has a way to keep the rate of false rejections at a controlled level, by definition via the frequentist long-run construction of tests. Here, "honest" means absence of uncontrolled biases, hidden selections, etc.

However, neither the robot, nor the agency have a personal stake in any particular drug or a part that goes through the assembly conveyor. In science, on the other hand, we, as individual investigators care most about the particular hypothesis we study, rather than about the proportion of spurious claims in our favorite journal we submit to. Neither the P-value magnitude nor the bounds of a confidence interval (CI) refer directly to our question about the credibility of what we report. When we construct the CI bounds, we should be saying that the only meaning of the two numbers is that if other scientists do the same kind of CI computation in their studies, the 95% or whatever coverage will be maintained over various studies as a whole.

In this light, I find it ironic that P-values are being "banned" by journals, considering that in the thick of replicability crisis they are of more value to journal editors than to researchers submitting their papers, as a practical way of keeping the rate of spurious findings reported by a journal at bay, in the long run. P-values are good at filtering, or as IJ Good wrote, they are good for protecting statistician's rear end, but not so much the rear end of the client.

P.S. I'm a huge fan of Benjamini and Hochberg's idea of taking the unconditional expectation across studies with multiple tests. Under the global "null", the "frequentist" FDR is still controlled - studies with one or more rejections pop up in a journal at a controlled rate, although, in this case, any study where some rejections have been actually made has the proportion of false rejections that is equal to one.

Answer 7

I agree with Matt that p-values are useful when the null hypothesis is true.

The simplest example I can think of is testing a random number generator. If the generator is working correctly, you can use any appropriate sample size of realizations and when testing the fit over many samples, the p-values should have a uniform distribution. If they do, this is good evidence for a correct implementation. If they don't, you know you have made an error somewhere.

Other similar situations occur when you know a statistic or random variable should have a certain distribution (again, the most obvious context is simulation). If the p-values are uniform, you have found support for a valid implementation. If not, you know you have a problem somewhere in your code.

Answer 8

I can think of example in which p-values are useful, in Experimental High Energy Physics. See Fig.1 This plot is taken from this paper: Observation of a new particle in the search for the Standard Model Higgs boson with the ATLAS detector at the LHC

In this Fig, the p-value is shown versus the mass of an hypothetical particle. The null hypothesis denotes the compatibility of the observation with a continuous background. The large ($5 \sigma$) deviation at m$_\mathrm{H} \approx 125$ GeV was the first evidence and discovery of a new particle. This earned François Englert, Peter Higgs the Nobel Prize in Physics in 2013.