3
$\begingroup$

I got the following quandary and wondering if you have any thoughts on how to show/test it.

I work with peer review data and notice that many of the most highly cited articles in my data were "first round consensus accept" decisions, as opposed to going through 2 or more rounds of review AND/OR being no consensus.

So I want to show/test whether the peer review process picked out the really good stuff (i.e. very high subsequent citations) especially effectively (i.e. first round consensus accept).

What's a good way to show this / what statistical test would I use?

The only way I've thought of so far -- and which I don't like much -- is the following: 1. rank all the articles by citations. 2. show that as include in your set lower and lower cited articles, the fraction of the group that were first-round-consensus-accept decreases. i.e. the top cited articles were disproportionately often first-round consensus accept.

fraction that's 1st-rd-cons-accept

Thanks in advance!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Create a Two By Two Chart:

Variable 1: Y=1 (First round acceptance), Y=0 (NO first Round Acceptance)

Variable 2: Z=1 (Citations > Median # of Citations), Z=0 (Citations>Median # of Citations)

enter image description here

Then calculate the marginal probabilities:

Pr (Z=1|Y=1) = The probability of having higher citations given first round acceptance

Pr (Z=1|Z=0) = The probability of having higher citations given NO first round acceptance

Compare the above conditional probabilities

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks @subra, makes sense, but I wonder if there isn't a more continuous way to do this than divide citations into two groups (above and below the median). $\endgroup$
    – DrMisha
    Commented Aug 6, 2015 at 0:28
  • $\begingroup$ @Misha: If you have more variables, you can run the following regression----> # of citations= β0+β1 X1+β2 X2+β3 1(if accepted in first round) You need β3 to have a positive coefficient. $\endgroup$
    – subra
    Commented Aug 6, 2015 at 2:57
  • 1
    $\begingroup$ Thanks @subra. That specification would tell me the (constant) effect on citations of being in group 1 vs. group 0. What I want is an effect the varies across the citation range.. showing that the higher the citations, the more likely the article to be a first-round-consensus-accept. One thing I'm pondering is a figure like the one I showed in the OP, but with a rolling window (of citations) on the x-axis... and "fraction 1st round consensus accept" on y-axis. Still don't know what statistical test I'd want to do though. $\endgroup$
    – DrMisha
    Commented Aug 6, 2015 at 16:30
  • 1
    $\begingroup$ @ Misha: If you did a quantile regression on above specification, it will give you the effect of first round acceptance on different quantiles of (# of citations.) This way you can see the effect varying at different levels of citations. $\endgroup$
    – subra
    Commented Aug 6, 2015 at 16:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.