This article "The Odds, Continually Updated" from NY Times happened to catch my attention. To be short, it states that

[Bayesian statistics] is proving especially useful in approaching complex problems, including searches like the one the Coast Guard used in 2013 to find the missing fisherman, John Aldridge (though not, so far, in the hunt for Malaysia Airlines Flight 370)........, Bayesian statistics are rippling through everything from physics to cancer research, ecology to psychology...

In the article, there are also some criticisms about the frequentist's p-value, for example:

Results are usually considered “statistically significant” if the p-value is less than 5 percent. But there is a danger in this tradition, said Andrew Gelman, a statistics professor at Columbia. Even if scientists always did the calculations correctly — and they don’t, he argues — accepting everything with a p-value of 5 percent means that one in 20 “statistically significant” results are nothing but random noise.

Besides above, perhaps the most famous paper criticizing p-value is this one - "Scientific method: Statistical errors" by Regina Nuzzo from Nature, in which a lot of scientific issues raised by p-value approach has been discussed, like reproducibility concerns, p-value hacking, etc.

P values, the 'gold standard' of statistical validity, are not as reliable as many scientists assume. ...... Perhaps the worst fallacy is the kind of self-deception for which psychologist Uri Simonsohn of the University of Pennsylvania and his colleagues have popularized the term P-hacking; it is also known as data-dredging, snooping, fishing, significance-chasing and double-dipping. “P-hacking,” says Simonsohn, “is trying multiple things until you get the desired result” — even unconsciously. ...... “That finding seems to have been obtained through p-hacking, the authors dropped one of the conditions so that the overall p-value would be less than .05”, and “She is a p-hacker, she always monitors data while it is being collected.”

Another thing is an interesting plot as following from here, with the comment about the plot:

No matter how small your effect may be, you can always do the hard work of gathering data in order to pass the threshold of p < .05. As long as the effect you're studying isn't non-existent, p-values just measure how much effort you've put into collecting data.

enter image description here

With all above, my questions are:

  1. What does Andrew Gelman's argument, in the second block quote, mean precisely? Why did he interpret 5-percent p-value as "one in 20 statistically significant results are noting but random noise"? I am not convinced since to me p-value is used to make inference on one single study. His point seems related to multiple testing.

    Update: Check Andrew Gelman's blog about this: No, I didn't say that! (Credits to @Scortchi, @whuber).

  2. Given the criticisms about p-value, and also given there are a lot of information criteria, like AIC, BIC, Mallow's $C_p$ for evaluating the significance of a model (hence variables), should we not use p-value for variable selection at all but use those model selection criteria?

  3. Are there any good practical guidances of using p-value for statistical analysis which could lead to more reliable research results?
  4. Would Bayesian modeling framework a better way to pursue, as some statistician advocate? Specifically, would Bayesian approach be more likely to resolve false finding or manipulating the data issues? I am not convinced here as well since the prior is very subjective in Bayesian approach. Are there any practical and well-known studies that show Bayesian approach is better than frequentist's p-value, or at least in some particular cases?

    Update: I would be particularly interested in whether there are cases that Bayesian approach is more reliable than frequentist's p-value approach. By "reliable", I mean the Bayesian approach is less likely to manipulate data for desired results. Any suggestions?


Update 6/9/2015

Just noticed the news, and thought it would be good to put it here for discussion.

Psychology journal bans P values

A controversial statistical test has finally met its end, at least in one journal. Earlier this month, the editors of Basic and Applied Social Psychology (BASP) announced that the journal would no longer publish papers containing P values because the statistics were too often used to support lower-quality research.

Along with a recent paper, "The fickle P value generates irreproducible results" from Nature, about P value.

Update 5/8/2016

Back in March, the American Statistical Association (ASA) released statements on statistical significance and p-values, "....The ASA statement is intended to steer research into a ‘post p<0.05 era.’"

This statement contains 6 principles that address the misuse of the p-value:

  1. P-values can indicate how incompatible the data are with a specified statistical model.
  2. P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.
  3. Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.
  4. Proper inference requires full reporting and transparency.
  5. A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.
  6. By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis.

Details: "The ASA's statement on p-values: context, process, and purpose".

  • 11
    Re 1: I suspect the Gelman block might be a misquotation, because it needs strong (counterfactual) assumptions to be correct. If everything ever studied in the world followed their null hypotheses, and all null hypotheses were simple (and not composite), then by construction 5% of all p-values less than $0.05$ would occur by chance--be "random noise." However, if people always performed detailed, extensive experiments where the alternative hypothesis is true (as in the last quotation), then essentially 100% of all p-values would be less than $0.05$ and none of them would be "noise." – whuber Jan 25 '15 at 20:40
  • 8
    @whuber: You're right: No, I didn’t say that!. – Scortchi Jan 25 '15 at 21:32
  • 3
    Good find, @Scortchi! For the record--in case the link ever goes bad--Gelman emphatically rejects the NY Times characterization (albeit very tactfully) and writes "accepting everything with a p-value of 5 percent can lead to spurious findings—cases where an observed “statistically significant” pattern in data does not reflect a corresponding pattern in the population—far more than 5 percent of the time." – whuber Jan 25 '15 at 21:34
  • 3
    In reference to your comment "As long as the effect you're studying isn't non-existent," that is the point of studies involving p values—to determine if the effect you're studying is truly present or if peculiarities in the data you've collected are just due to random chance. Lowering the p value with increasing sample size is completely mathematically sound and, in fact, the only option. You're in no way "hacking" the p-value. From an intuitive standpoint, it makes sense that putting more effort into collecting data would translate into a higher confidence in the conclusions you draw from it. – David Webb Jan 25 '15 at 23:54
  • 1
    @DavidWebb Agreed. If the effect size is small, that's okay and it will be easier to say how large or small the effect is with more data. If you can get more data, you should. – Desty Jan 26 '15 at 10:53

Here are some thoughts:

  1. As @whuber notes, I doubt Gelman said that (although he may have said something similar sounding). Five percent of cases where the null is true will yield significant results (type I errors) using an alpha of .05. If we assume that the true power for all studies where the null was false were $80\%$, the statement could only be true if the ratio of studies undertaken where the null was true to studies in which the null was false was $100/118.75 \approx 84\%$.
  2. Model selection criteria, such as the AIC, can be seen as a way of selecting an appropriate $p$-value. To understand this more fully, it may help to read @Glen_b's answer here: Stepwise regression in R – Critical p-value. Moreover, nothing prevents people from 'AIC-hacking', if the AIC became the requirement for publication.
  3. A good guide to fitting models in such a manner that you don't invalidate your $p$-values would be Frank Harrell's book, Regression Modeling Strategies.
  4. I am not dogmatically opposed to using Bayesian methods, but I do not believe they would solve this problem. For example, you can just keep collecting data until the credible interval no longer included whatever value you wanted to reject. Thus you have 'credible interval-hacking'. As I see it, the issue is that many practitioners are not intrinsically interested in the statistical analyses they use, so they will use whichever method is required of them in an unthinking and mechanical way. For more on my perspective here, it may help to read my answer to: Effect size as the hypothesis for significance testing.
  • 10
    (+1) An easy way to hack a credible interval is to adopt just the right prior :-). Of course no competent practitioner would do this--Gelman emphasizes using sensitivity assessments, uninformative hyperpriors, etc.--but then again no competent user of hypothesis tests would do p-value hacking, would they? On the other hand, in a Bayesian analysis it might be more difficult to hide what one is doing--assuming the prior is clearly disclosed--compared to all the undocumented analyses that may be involved in p-value hacking. – whuber Jan 25 '15 at 21:32
  • 1
    @whuber, that's true, but I think we can set aside any issues w/ the inappropriateness or subjectivity of the prior. If the true effect isn't exactly 0, w/ enough data the credible interval will eventually not include 0, just as the p will be <.05 (cf, the last quote), so you can just keep collecting data until you get the result you want irrespective of the prior. – gung Jan 25 '15 at 21:38
  • 4
    Good points. I am reminded of a recent question about predicting failures in 10,000 products after observing no failures in 100,000 of them. The answer is pretty sensitive to the prior because failures are so rare. This may be the kind of exceptional situation that "proves the rule"; it shows that in reality it can be impracticable to collect enough data to obtain a desired result. That's exactly when some clients start imploring the statistician to "do their magic" to achieve the desired outcome! Probably many readers have felt that pressure before ... . – whuber Jan 25 '15 at 21:43
  • 1
    @gung, in practical clinic trials, there are always stopping criteria at different phases for recruiting more subjects for experiments. In that sense, would Bayesian approach sound less likely to manipulate the credible interval thus the research conclusions? – Aaron Zeng Jan 25 '15 at 23:25
  • 1
    @AaronZeng, it seems to me that explicit stopping criteria apply equally to Frequentist & Bayesian perspectives. I don't see any net advantage / disadvantage here. – gung Jan 25 '15 at 23:53

To me, one of the most interesting things about the p-hacking controversy is that the entire history of p<=0.05 as the "once in a blue moon" standard for statistical significance, as Joseph Kaldane noted in a JASA article on forensic statistics back in the 90s, rests on absolutely no statistical theory whatsoever. It's a convention, simple heuristic and rule of thumb that started with R.A. Fisher and has since been reified or consecrated into its present "unquestioned" status. Bayesian or not, the time is long overdue to challenge this metric standard or at least give it the skepticism it deserves.

That said, my interpretation of Gelman's point is that, as is well known, the peer review process rewards positive statistical significance and punishes insignificant results by not publishing those papers. This is irrespective of whether or not publishing an insignificant finding would have potentially large impact on the thinking and theorizing for a given domain. Gelman, Simonshohn and others have repeatedly pointed to the abuse of the 0.05 significance level in peer-reviewed and published research by holding up examples of ridiculous, yet statistically significant findings in paranormal, social and psychological research. One of the most egregious was the statistically significant finding that pregnant women were more likely to wear red dresses. Gelman maintains that, in the absence of logical challenges to statistical results, the mere fact that an analysis is "statistically significant" is a potentially meaningless explanation. Here, he's referring to the industry's occupational hazard with overly technical and abstruse arguments that do little or nothing to advance a debate among a lay audience.

This is a point Gary King makes vehemently when he practically begs quantitative political scientists (and, by extension, all quants) to stop mechanistic, technical reportage such as "this result was significant at a p<=0.05 level" and moving towards more substantive interpretations. Here's a quote from a paper by him,

(1) convey numerically precise estimates of the quantities of greatest substantive interest, (2) include reasonable measures of uncertainty about those estimates, and (3) require little specialized knowledge to understand. The following simple statement satisfies our criteria: 'Other things being equal, an additional year of education would increase your annual income by 1,500 dollars on average, plus or minus about 500 dollars.' Any smart high school student would understand that sentence, no matter how sophisticated the statistical model and powerful the computers used to produce it.

King's point is very well taken and maps out the direction the debate needs to take.

Making the Most of Statistical Analyses: Improving Interpretation and Presentation, King, Tomz and Wittenberg, 2002, Am Jour of Poli Sci.

  • 1
    +1 Thank you for this readable, informative, and thoughtful contribution to the thread. – whuber Jun 9 '15 at 16:43
  • @whuber Thanks for the kind words. Time will tell if other participants agree with it or not. – Mike Hunter Jun 9 '15 at 16:48
  • 1
    I may be deluded, but I like to think that some (if not most) of our active voters do not vote on the basis of agreement or disagreement, but on whether a post responds to the original question in a way that is clear and authoritative. After all, the hover text above the upvote icon reads "This answer is useful," not "I agree with this guy." (This is not to be confused with voting on our meta site, which does signify degree of agreement.) Some evidence for this impression is afforded by the many sportsmanship badges awarded. – whuber Jun 9 '15 at 16:54
  • @Whuber The nuance you point out is duly noted. – Mike Hunter Jun 9 '15 at 17:21
  • @whuber this thread was the source of my use of the word deluded in our chat the other day. – Mike Hunter Jun 13 '15 at 10:38

Here are some of my thoughts regarding Question 3 after reading all the insightful comments and answers.

Perhaps one practical guidance in statistical analysis to avoid p-value hacking is to instead look at the scientifically (or, biologically, clinically, etc) significant/meaningful effect size.

Specifically, the research should pre-define the effect size that can be declared useful or meaningful before the data analysis or even before the data collection. For example, if let $\theta$ denote a drug effect, instead of testing the following hypothesis, $$H_0: \theta = 0 \quad \quad vs. \quad \quad H_a: \theta \neq 0,$$ one should always test $$H_0: \theta < \delta \quad \quad vs. \quad \quad H_a: \theta \ge \delta,$$ with $\delta$ being the predefined effect size to claim meaningful significance.

In addition, to avoid of using too large sample size to detect the effect, the sample size required should be taken into account as well. That is, we should put a constrain on the maximum sample size used for the experiment.

To sum up,

  1. We need predefine a threshold for the meaningful effect size to declare significance;
  2. We need to predefine a threshold for sample size used in the experiment to quantify how detectable the meaningful effect size is;

With above, maybe we can therefore avoid minor "significant" effect claimed by a huge sample size.


[Update 6/9/2015]

Regarding Question 3, here are some suggestions based on the recent paper from nature: "The fickle P value generates irreproducible results" as I mentioned in the Question part.

  1. Report effect size estimates and their precision, i.e. 95% confidence interval, since those more informative information answer exactly questions like how big is the difference, or how strong is the relationship or association;
  2. Put the effect size estimates and 95% CIs into the context of the specific scientific studies/questions and focus on their relevance of answering those questions and discount the fickle P value;
  3. Replace the power analysis with "planning for precision" to determine the sample size required for estimating the effect size to reach a defined degree of precision.

[End update 6/9/2015]

  • 4
    If you rewrite $H_0: \theta = \delta$ then you are arguing for equivalence testing, which I think is a fine thing to do in many situations. (Typically hypothesis tests are not presented like the second situation, because there are potential outcomes not in the null or alternative.) – Andy W Jan 26 '15 at 21:26
  • @AndyW, Thanks for the comments. I've changed my answer accordingly. Would that sound a better option? – Aaron Zeng Jan 26 '15 at 21:41
  • 2
    +1 for the reference to that Nature article. It contains some astonishing misinformation, though, such as the (unannounced) Bayesian interpretation of p-values: "As an example, if a study obtains P = 0.03, there is a 90% chance that a replicate study would return a P value somewhere between the wide range of 0–0.6 (90% prediction intervals), whereas the chances of P < 0.05 is just 56%." I wonder what prior distribution the authors are assuming--and why that's even relevant? – whuber Jun 9 '15 at 16:24
  • @AndyW and Aaron Zeng, even better is to combine results from both tests for difference, and tests for equivalence. That way, one places both relevant effect size and statistical power explicitly into the conclusions one draws (see the section on relevance tests). – Alexis Feb 17 at 19:58

In contemporary usage the p-value refers to the cumulative probability of the data given the null hypothesis being at or greater than some threshold. I.e. $P(D|H_0)\le\alpha$. I think that $H_0$ tends to be a hypothesis of 'no effect' usually proxied by a comparison to the probability to a satisfactorily unlikely random result in some number of trials. Dependent on the field it varies from 5% down to 0.1% or less. However, $H_0$ does not have to be a comparison to random.

  1. It implies that 1/20 results may reject the null when they should not have. If science based it's conclusion on single experiments then the statement would be defensible. Otherwise, if experiments were repeatable it would imply that 19/20 would not be rejected. The moral of the story is that experiments should be repeatable.

  2. Science is a tradition grounded in "objectivity" so "objective probability" naturally appeals. Recall that experiments are suppose to demonstrate a high degree of control often employing block design and randomisation to control for factors outside of study. Thus, comparison to random does make sense because all other factors are supposed to be controlled for except for the ones under study. These techniques were highly successful in agriculture and industry prior to being ported to science.

  3. I'm not sure if a lack of information was ever really the problem. It's notable that for many in the non-mathematical sciences that statistics is just a box to tick.

  4. I'd suggest a general read about decision theory which unites the two frameworks. It simply comes down to using as much information as you have. Frequentist statistics assume parameters in models have unknown values from fixed distributions. Bayesians assume parameters in models come from distributions conditioned by what we know. If there is enough information to form a prior and enough information to update it to an accurate posterior then that's great. If there isn't then you may end up with worse results.

Reproducibility of statistical test results

This is a short, simple exercise to assess the reproducibility of decisions based on statistical testing.

Consider a null hypothesis H0 with a set of alternative hypotheses containing H1 and H2. Setup the statistical hypothesis test procedure at a significance level of 0.05 to have a power of 0.8, if H1 is true. Further assume that the power for H2 is 0.5. To assess reproducibility of test result, the experiment is considered of executing the test procedure two times. Starting with the situation, where H0 is true, the probabilities for the outcomes of the joint experiment are displayed in Table 1. The probability of not being able to reproduce decisions is 0.095.

Table 1. Frequencies, if H0 is true

\begin{array} {|r|r|} \hline Frequency. of. decision &Reject. H0 &Retain. H0 \\ \hline Reject. H0 &0.0025 &0.0475 \\ \hline Retain. H0 &0.0475 &0.9025 \\ \hline \end{array}

The frequencies change as the true state of nature changes. Assuming H1 is true, H0 can be rejected as designed with a power of 0.8. The resulting frequencies for the different outcomes of the joint experiment are displayed in Table 2. The probability of not being able to reproduce decisions is 0.32.

Table 2. Frequencies, if H1 is true

\begin{array} {|r|r|} \hline Frequency. of. decision &Reject. H0 &Retain. H0 \\ \hline Reject. H0 &0.64 &0.16 \\ \hline Retain. H0 &0.16 &0.04 \\ \hline \end{array}

Assuming H2 is true, H0 will be rejected with a probability of 0.5. The resulting frequencies for the different outcomes of the joint experiment are displayed in Table 3. The probability of not being able to reproduce decisions is 0.5.

Table 3. Frequencies, if H2 is true

\begin{array} {|r|r|} \hline Frequency. of. decision &Reject. H0 &Retain. H0 \\ \hline Reject. H0 &0.25 &0.25 \\ \hline Retain. H0 &0.25 &0.25 \\ \hline \end{array}

The test procedure was designed to control type I errors (the rejection of the null hypothesis even though it is true) with a probability of 0.05 and limit type II errors (no rejection of the null hypothesis even though it is wrong and H1 is true) to 0.2. For both cases, with either H0 or H1 assumed to be true, this leads to non-negligible frequencies, 0.095 and 0.32, respectively, of "non-reproducible", "contradictory" decisions, if the same experiment is repeated twice. The situation gets worse with a frequency up to 0.5 for "non-reproducible", "contradictory" decisions, if the true state of nature is between the null- and the alternative hypothesis used to design the experiment.

The situation can also get better - if type 1 errors are controlled more strictly, or if the true state of nature is far away from the null, which results in a power to reject the null that is close to 1.

Thus, if you want more reproducible decisions, increase the significance level and the power of your tests. Not very astonishing ...

  • (+1) But you can't set the p-value to 5% before the experiment - think you mean "significance level". – Scortchi Jan 18 '16 at 13:43
  • Thank you. Same thing in the last sentence: "decrease the significance levels and increase the power" – Scortchi Jan 18 '16 at 13:53

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.