Comparisons of two rankings There are two methods which produce different ranking of a group of hypotheses.
I want to compare the two methods showing that one method generally produces
lower ranking than the other method for a specific subset of the considered group of 
hypotheses. I was thinking of comparing the quantiles of the two rankings on this subset,
but not sure whether that was the right thing to do. Can anyone familiar with this give 
some hint? Thanks. Hanna
 A: There may be a more test for this, but I'd say a good start is simply tallying up the number of rankings which are lower, equal to, and greater for one method vs. the other.
E.g. if you have points $i=1,2,3...I$ and rankings $A_i$ and $B_i$ for every point, then make a table:
$A>B$: 15
$A=B$: 5
$A<B$: 3
(The actual counts 15, 5, and 3 are obviously only examples.)
You'll see the pattern pretty quickly.  A test given that table shouldn't be hard to devise (a t-test of counts $A>B$ vs. $A<B$ might do it, but there's likely something sticky in the fact that A>B is mechanistically related to B>A).
Note that you're discarding information this way versus the actual rankings, so there's likely a more efficient method out there.
Test
Here's a labeling test.  I think but an by no means certain that this is a general test for A and B being drawn from the same distribution, but it could be somewhat more specific than that.
set.seed(100)
S <- seq(20)
A <- sample(S) # Replace with your actual rankings
B <- sample(S) # Replace with your actual rankings


ci <- quantile( replicate( 1000, sum(sample(S)>sample(S)) ), p=c(.025,.975))

sum(A>B)    

> ci
 2.5% 97.5% 
    7    12 
> sum(A>B)
[1] 8

Since the count of A>B is within the CI, we cannot reject the null.
A: This looks like a straightforward application of the binomial test, with the two categories being "method A ranks this hypothesis higher than method B" and "method B ranks this hypothesis higher than method A" (discarding ties), applied to all the hypotheses in your subset.
