# Real world performance of p<0.005 compared with 0.05

Within scientific literature, there is a tentative proposal to change the significance level from p=0.05 to p=0.005

http://www.nature.com/articles/s41562-017-0189-z

I understand there is a lot of nuance to this proposal and don't necessarily want to get too much into the pros and cons!

In order to test the performance of this proposal in the real world, I have collated primary endpoint p values for a large number of scientific studies, and have assigned liekert-score type ordinal data to describe the value of the study, where 1= low importance study and 5= highly important study (based on a complex calculation taking journal impact factor, number of citations, H-score and a few other factors into account).

So I have two columns of data as follows:

Column A: 1-5 (Ordinal) - Where 1 = Low Importance Study; 5 = High Importance Study

Column B: 0-1 (Categorical, Dichotomous - Does the study primary endpoint meet p= <0.005 YES or NO - Represented as 1 or 0)

I can see visually from the data that the primary endpoint of most low value studies does not meet the significance threshold of p=0.005 (20%), but most high value studies do meet it (89%). Using the breakdown from studies I have analysed so far, the breakdown according to likert scale is as follows:

1: 20% meet p <0.005

2: 63% meet p <0.005

3: 85% meet p <0.005

4: 93% meet p <0.005

5: 89% meet p <0.005

If I group studies scoring 1-2 as "Not valuable", and studies scoring 3-4-5 as "Moderately/Very Valuable", I get:

1-2: 45% meet p <0.005

3-4-5: 89% meet p <0.005

I am wondering how I can describe this better in statistical terms, and what test would be appropriate here to describe the association with study value and the binary metric of meeting p = <0.005. In laymans terms, I would like to describe the efficiency of this new threshold at identifying and excluding low quality papers, as well as its performance in identifying but preserving high quality papers.

Is Spearman's rho appropriate here? Or would I be better off trying to describe this using receiver operating curves and with the language of sensitivity and specificity etc?

For interest, my data is here https://ufile.io/k3abnh1s

• How would your analysis account for publication bias and p-hacking? It isn't as if these p values are all reported faithfully. Commented Jan 9, 2022 at 6:40
• 1. While I agree that often 5% is much too low (but sometimes not), the very notion of "a" significance level for "science" seems strange to me, is likely risible to your average Bayesian, and surely would be a bad joke to the 5-sigma physicists. 2. Aren't some of the items in your valuableness score likely in part determined by p-value? Commented Jan 9, 2022 at 6:57
• ... i.e. measuring the extent to which low p-values are associated with low p-values. 3. Note that a test does not describe association. If you have a hypothesis, name it. If you don't, don't think in terms of tests (e.g. maybe you're after estimation or perhaps even just diagnostics) . Commented Jan 9, 2022 at 7:08
• As an example of @Glen_b's "(but sometimes not)", when comparing the performance of novel machine learning algorithms against state-of-the-art baselines (e.g. Random Forest, Support Vector Machines, etc.) p < 0.05 is likely to stifle research as you wouldn't expect to be significantly better than the state-of-the-art with very stringent significance levels. The significance level should be set according to the needs of the analysis and we shouldn't use defaults values without thinking about what is appropriate. Commented Jan 9, 2022 at 12:02
• @Alexis I think we may be talking at cross purposes. I don't think I said that "results of a hypothesis test are not evidence for or against association in a general sense". There was a point where I was talking about the distinction between (i) describing association - which doesn't in any way require a hypothesis (but in which descriptive statistics would play a role - such as plots or measures of association, for example) and (ii) testing it, in which a hypothesis is definitely required. (If I have misunderstood your intent there, I apologize.) ... Commented Jan 9, 2022 at 12:18

There is a straight test for equality of several proportions, see for example prop.test in R (or in fact chisq.test). There is also some work about testing for monotonicity (if you want to test that proportions go up as a function of a Likert score), see Sec. 6 of Mervyn J. Silvapulle, Pranab K. Sen, "Constrained Statistical Inference: Inequality, Order, and Shape Restrictions", Wiley 2001, although I'm not sure whether software is available (a comment mentions this: https://restriktor.org/).