Real world performance of p<0.005 compared with 0.05 Within scientific literature, there is a tentative proposal to change the significance level from p=0.05 to p=0.005
http://www.nature.com/articles/s41562-017-0189-z
I understand there is a lot of nuance to this proposal and don't necessarily want to get too much into the pros and cons!
In order to test the performance of this proposal in the real world, I have collated primary endpoint p values for a large number of scientific studies, and have assigned liekert-score type ordinal data to describe the value of the study, where 1= low importance study and 5= highly important study (based on a complex calculation taking journal impact factor, number of citations, H-score and a few other factors into account).
So I have two columns of data as follows:
Column A: 1-5 (Ordinal) - Where 1 = Low Importance Study; 5 = High Importance Study
Column B: 0-1 (Categorical, Dichotomous - Does the study primary endpoint meet p= <0.005 YES or NO - Represented as 1 or 0)
I can see visually from the data that the primary endpoint of most low value studies does not meet the significance threshold of p=0.005 (20%), but most high value studies do meet it (89%). Using the breakdown from studies I have analysed so far, the breakdown according to likert scale is as follows:
1: 20% meet p <0.005
2: 63% meet p <0.005
3: 85% meet p <0.005
4: 93% meet p <0.005
5: 89% meet p <0.005
If I group studies scoring 1-2 as "Not valuable", and studies scoring 3-4-5 as "Moderately/Very Valuable", I get:
1-2: 45% meet p <0.005
3-4-5: 89% meet p <0.005
I am wondering how I can describe this better in statistical terms, and what test would be appropriate here to describe the association with study value and the binary metric of meeting p = <0.005. In laymans terms, I would like to describe the efficiency of this new threshold at identifying and excluding low quality papers, as well as its performance in identifying but preserving high quality papers.
Is Spearman's rho appropriate here? Or would I be better off trying to describe this using receiver operating curves and with the language of sensitivity and specificity etc?
For interest, my data is here https://ufile.io/k3abnh1s
 A: There is a straight test for equality of several proportions, see for example prop.test in R (or in fact chisq.test). There is also some work about testing for monotonicity (if you want to test that proportions go up as a function of a Likert score), see Sec. 6 of Mervyn J. Silvapulle, Pranab K. Sen, "Constrained Statistical Inference: Inequality, Order, and Shape Restrictions", Wiley 2001, although I'm not sure whether software is available (a comment mentions this: https://restriktor.org/).
One could in principle also fit a logistic regression model, but in my view it is more relevant here whether proportions can be monotonic (which is assumed rather than tested by a logistic regression) rather than estimating a regression coefficient.
That said, as you have no information about which of the tested null hypotheses are really true (or at least true in the sense of "scientific approximation" - in fact no formal statistical null hypothesis is literally true) and which are not, I don't see how these data can tell you anything about "performance" or "efficiency". Surely this is not measured by your importance score in any sense. (I'm not denying that your data and the question they could address may be interesting in their own right anyway, but chances are if done seriously this would require the incorporation of more information such as sample sizes, number of tests performed per study etc.)
I'd by the way advise against trying to do something sophisticated for the sake of it. Your table of probability vs. Likert score plus numbers of observations summarises the data properly. An equality or monotonicity test may tell you something worthwhile on top of that, but I don't really see the added value of Spearman, logistic regression, receiver operating curves etc. here.
