Is p-value essentially useless and dangerous to use?

Question

This article "The Odds, Continually Updated" from NY Times happened to catch my attention. To be short, it states that

[Bayesian statistics] is proving especially useful in approaching complex problems, including searches like the one the Coast Guard used in 2013 to find the missing fisherman, John Aldridge (though not, so far, in the hunt for Malaysia Airlines Flight 370)........, Bayesian statistics are rippling through everything from physics to cancer research, ecology to psychology...

In the article, there are also some criticisms about the frequentist's p-value, for example:

Results are usually considered “statistically significant” if the p-value is less than 5 percent. But there is a danger in this tradition, said Andrew Gelman, a statistics professor at Columbia. Even if scientists always did the calculations correctly — and they don’t, he argues — accepting everything with a p-value of 5 percent means that one in 20 “statistically significant” results are nothing but random noise.

Besides above, perhaps the most famous paper criticizing p-value is this one - "Scientific method: Statistical errors" by Regina Nuzzo from Nature, in which a lot of scientific issues raised by p-value approach has been discussed, like reproducibility concerns, p-value hacking, etc.

P values, the 'gold standard' of statistical validity, are not as reliable as many scientists assume. ...... Perhaps the worst fallacy is the kind of self-deception for which psychologist Uri Simonsohn of the University of Pennsylvania and his colleagues have popularized the term P-hacking; it is also known as data-dredging, snooping, fishing, significance-chasing and double-dipping. “P-hacking,” says Simonsohn, “is trying multiple things until you get the desired result” — even unconsciously. ...... “That finding seems to have been obtained through p-hacking, the authors dropped one of the conditions so that the overall p-value would be less than .05”, and “She is a p-hacker, she always monitors data while it is being collected.”

Another thing is an interesting plot as following from here, with the comment about the plot:

No matter how small your effect may be, you can always do the hard work of gathering data in order to pass the threshold of p < .05. As long as the effect you're studying isn't non-existent, p-values just measure how much effort you've put into collecting data.

With all above, my questions are:

1. What does Andrew Gelman's argument, in the second block quote, mean precisely? Why did he interpret 5-percent p-value as "one in 20 statistically significant results are noting but random noise"? I am not convinced since to me p-value is used to make inference on one single study. His point seems related to multiple testing.

   Update: Check Andrew Gelman's blog about this: No, I didn't say that! (Credits to @Scortchi, @whuber).

2. Given the criticisms about p-value, and also given there are a lot of information criteria, like AIC, BIC, Mallow's $C_p$ for evaluating the significance of a model (hence variables), should we not use p-value for variable selection at all but use those model selection criteria?

3. Are there any good practical guidances of using p-value for statistical analysis which could lead to more reliable research results?

4. Would Bayesian modeling framework a better way to pursue, as some statistician advocate? Specifically, would Bayesian approach be more likely to resolve false finding or manipulating the data issues? I am not convinced here as well since the prior is very subjective in Bayesian approach. Are there any practical and well-known studies that show Bayesian approach is better than frequentist's p-value, or at least in some particular cases?

   Update: I would be particularly interested in whether there are cases that Bayesian approach is more reliable than frequentist's p-value approach. By "reliable", I mean the Bayesian approach is less likely to manipulate data for desired results. Any suggestions?

---

Update 6/9/2015

Just noticed the news, and thought it would be good to put it here for discussion.

Psychology journal bans P values

A controversial statistical test has finally met its end, at least in one journal. Earlier this month, the editors of Basic and Applied Social Psychology (BASP) announced that the journal would no longer publish papers containing P values because the statistics were too often used to support lower-quality research.

Along with a recent paper, "The fickle P value generates irreproducible results" from Nature, about P value.

Update 5/8/2016

Back in March, the American Statistical Association (ASA) released statements on statistical significance and p-values, "....The ASA statement is intended to steer research into a ‘post p<0.05 era.’"

This statement contains 6 principles that address the misuse of the p-value:

1. P-values can indicate how incompatible the data are with a specified statistical model.
2. P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.
3. Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.
4. Proper inference requires full reporting and transparency.
5. A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.
6. By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis.

Details: "The ASA's statement on p-values: context, process, and purpose".

Answer 1

Here are some thoughts:

1. As @whuber notes, I doubt Gelman said that (although he may have said something similar sounding). Five percent of cases where the null is true will yield significant results (type I errors) using an alpha of .05. If we assume that the true power for all studies where the null was false were $80\%$, the statement could only be true if the ratio of studies undertaken where the null was true to studies in which the null was false was $100/118.75 \approx 84\%$.
2. Model selection criteria, such as the AIC, can be seen as a way of selecting an appropriate $p$-value. To understand this more fully, it may help to read @Glen_b's answer here: Stepwise regression in R – Critical p-value. Moreover, nothing prevents people from 'AIC-hacking', if the AIC became the requirement for publication.
3. A good guide to fitting models in such a manner that you don't invalidate your $p$-values would be Frank Harrell's book, Regression Modeling Strategies.
4. I am not dogmatically opposed to using Bayesian methods, but I do not believe they would solve this problem. For example, you can just keep collecting data until the credible interval no longer included whatever value you wanted to reject. Thus you have 'credible interval-hacking'. As I see it, the issue is that many practitioners are not intrinsically interested in the statistical analyses they use, so they will use whichever method is required of them in an unthinking and mechanical way. For more on my perspective here, it may help to read my answer to: Effect size as the hypothesis for significance testing.

Answer 2

To me, one of the most interesting things about the p-hacking controversy is that the entire history of p<=0.05 as the "once in a blue moon" standard for statistical significance, as Joseph Kaldane noted in a JASA article on forensic statistics back in the 90s, rests on absolutely no statistical theory whatsoever. It's a convention, simple heuristic and rule of thumb that started with R.A. Fisher and has since been reified or consecrated into its present "unquestioned" status. Bayesian or not, the time is long overdue to challenge this metric standard or at least give it the skepticism it deserves.

That said, my interpretation of Gelman's point is that, as is well known, the peer review process rewards positive statistical significance and punishes insignificant results by not publishing those papers. This is irrespective of whether or not publishing an insignificant finding would have potentially large impact on the thinking and theorizing for a given domain. Gelman, Simonshohn and others have repeatedly pointed to the abuse of the 0.05 significance level in peer-reviewed and published research by holding up examples of ridiculous, yet statistically significant findings in paranormal, social and psychological research. One of the most egregious was the statistically significant finding that pregnant women were more likely to wear red dresses. Gelman maintains that, in the absence of logical challenges to statistical results, the mere fact that an analysis is "statistically significant" is a potentially meaningless explanation. Here, he's referring to the industry's occupational hazard with overly technical and abstruse arguments that do little or nothing to advance a debate among a lay audience.

This is a point Gary King makes vehemently when he practically begs quantitative political scientists (and, by extension, all quants) to stop mechanistic, technical reportage such as "this result was significant at a p<=0.05 level" and moving towards more substantive interpretations. Here's a quote from a paper by him,

(1) convey numerically precise estimates of the quantities of greatest substantive interest, (2) include reasonable measures of uncertainty about those estimates, and (3) require little specialized knowledge to understand. The following simple statement satisfies our criteria: 'Other things being equal, an additional year of education would increase your annual income by 1,500 dollars on average, plus or minus about 500 dollars.' Any smart high school student would understand that sentence, no matter how sophisticated the statistical model and powerful the computers used to produce it.

King's point is very well taken and maps out the direction the debate needs to take.

Making the Most of Statistical Analyses: Improving Interpretation and Presentation, King, Tomz and Wittenberg, 2002, Am Jour of Poli Sci.

Answer 3

Here are some of my thoughts regarding Question 3 after reading all the insightful comments and answers.

Perhaps one practical guidance in statistical analysis to avoid p-value hacking is to instead look at the scientifically (or, biologically, clinically, etc) significant/meaningful effect size.

Specifically, the research should pre-define the effect size that can be declared useful or meaningful before the data analysis or even before the data collection. For example, if let $\theta$ denote a drug effect, instead of testing the following hypothesis, $$H_0: \theta = 0 \quad \quad vs. \quad \quad H_a: \theta \neq 0,$$ one should always test $$H_0: \theta < \delta \quad \quad vs. \quad \quad H_a: \theta \ge \delta,$$ with $\delta$ being the predefined effect size to claim meaningful significance.

In addition, to avoid of using too large sample size to detect the effect, the sample size required should be taken into account as well. That is, we should put a constrain on the maximum sample size used for the experiment.

To sum up,

1. We need predefine a threshold for the meaningful effect size to declare significance;
2. We need to predefine a threshold for sample size used in the experiment to quantify how detectable the meaningful effect size is;

With above, maybe we can therefore avoid minor "significant" effect claimed by a huge sample size.

---

[Update 6/9/2015]

Regarding Question 3, here are some suggestions based on the recent paper from nature: "The fickle P value generates irreproducible results" as I mentioned in the Question part.

1. Report effect size estimates and their precision, i.e. 95% confidence interval, since those more informative information answer exactly questions like how big is the difference, or how strong is the relationship or association;
2. Put the effect size estimates and 95% CIs into the context of the specific scientific studies/questions and focus on their relevance of answering those questions and discount the fickle P value;
3. Replace the power analysis with "planning for precision" to determine the sample size required for estimating the effect size to reach a defined degree of precision.

[End update 6/9/2015]

Answer 4

In contemporary usage the p-value refers to the cumulative probability of the data given the null hypothesis being at or greater than some threshold. I.e. $P(D|H_0)\le\alpha$. I think that $H_0$ tends to be a hypothesis of 'no effect' usually proxied by a comparison to the probability to a satisfactorily unlikely random result in some number of trials. Dependent on the field it varies from 5% down to 0.1% or less. However, $H_0$ does not have to be a comparison to random.

1. It implies that 1/20 results may reject the null when they should not have. If science based it's conclusion on single experiments then the statement would be defensible. Otherwise, if experiments were repeatable it would imply that 19/20 would not be rejected. The moral of the story is that experiments should be repeatable.

2. Science is a tradition grounded in "objectivity" so "objective probability" naturally appeals. Recall that experiments are suppose to demonstrate a high degree of control often employing block design and randomisation to control for factors outside of study. Thus, comparison to random does make sense because all other factors are supposed to be controlled for except for the ones under study. These techniques were highly successful in agriculture and industry prior to being ported to science.

3. I'm not sure if a lack of information was ever really the problem. It's notable that for many in the non-mathematical sciences that statistics is just a box to tick.

4. I'd suggest a general read about decision theory which unites the two frameworks. It simply comes down to using as much information as you have. Frequentist statistics assume parameters in models have unknown values from fixed distributions. Bayesians assume parameters in models come from distributions conditioned by what we know. If there is enough information to form a prior and enough information to update it to an accurate posterior then that's great. If there isn't then you may end up with worse results.

Answer 5

Reproducibility of statistical test results

This is a short, simple exercise to assess the reproducibility of decisions based on statistical testing.

Consider a null hypothesis H0 with a set of alternative hypotheses containing H1 and H2. Setup the statistical hypothesis test procedure at a significance level of 0.05 to have a power of 0.8, if H1 is true. Further assume that the power for H2 is 0.5. To assess reproducibility of test result, the experiment is considered of executing the test procedure two times. Starting with the situation, where H0 is true, the probabilities for the outcomes of the joint experiment are displayed in Table 1. The probability of not being able to reproduce decisions is 0.095.

Table 1. Frequencies, if H0 is true

\begin{array} {|r|r|} \hline Frequency. of. decision &Reject. H0 &Retain. H0 \\ \hline Reject. H0 &0.0025 &0.0475 \\ \hline Retain. H0 &0.0475 &0.9025 \\ \hline \end{array}

The frequencies change as the true state of nature changes. Assuming H1 is true, H0 can be rejected as designed with a power of 0.8. The resulting frequencies for the different outcomes of the joint experiment are displayed in Table 2. The probability of not being able to reproduce decisions is 0.32.

Table 2. Frequencies, if H1 is true

\begin{array} {|r|r|} \hline Frequency. of. decision &Reject. H0 &Retain. H0 \\ \hline Reject. H0 &0.64 &0.16 \\ \hline Retain. H0 &0.16 &0.04 \\ \hline \end{array}

Assuming H2 is true, H0 will be rejected with a probability of 0.5. The resulting frequencies for the different outcomes of the joint experiment are displayed in Table 3. The probability of not being able to reproduce decisions is 0.5.

Table 3. Frequencies, if H2 is true

\begin{array} {|r|r|} \hline Frequency. of. decision &Reject. H0 &Retain. H0 \\ \hline Reject. H0 &0.25 &0.25 \\ \hline Retain. H0 &0.25 &0.25 \\ \hline \end{array}

The test procedure was designed to control type I errors (the rejection of the null hypothesis even though it is true) with a probability of 0.05 and limit type II errors (no rejection of the null hypothesis even though it is wrong and H1 is true) to 0.2. For both cases, with either H0 or H1 assumed to be true, this leads to non-negligible frequencies, 0.095 and 0.32, respectively, of "non-reproducible", "contradictory" decisions, if the same experiment is repeated twice. The situation gets worse with a frequency up to 0.5 for "non-reproducible", "contradictory" decisions, if the true state of nature is between the null- and the alternative hypothesis used to design the experiment.

The situation can also get better - if type 1 errors are controlled more strictly, or if the true state of nature is far away from the null, which results in a power to reject the null that is close to 1.

Thus, if you want more reproducible decisions, increase the significance level and the power of your tests. Not very astonishing ...