# Test to compare rank-ordered data

I have two result sets of identical search queries run under different parameters, analysis, etc. I will refer to them as RS1 and RS2. RS1 and RS2 may not have the same length, but they are rank-ordered with the first entry in the set being the highest priority. If RS1 is the "desired" set, or perhaps "control set", what statistical measures can I employ to determine how far off RS2 is from RS1?

For example, if the result sets return user ids:

RS1: [2, 4, 6], RS2: [4, 6, 2]

We can see that RS1 and RS2 are decently similar by inspection.

But in this example:

RS1: [2, 3, 4], RS2: [2, 6, 7]

is a much "worse" match. Obviously preferences of how important order is, etc matter, but what will be a good starting point? Any resources I could check out?

• I have heard of Jaccard index, but I think the measure needs to be more complicated than that since rank matters. Jun 20, 2017 at 21:13

You can compare the medians of the two distributions via Mann-Whitney test.

The test in R is implemented in the wilcox.test function in base R.

The R code to do perform both of these tests is:

n<-100
y<- rgamma(n, shape=1, scale=1)
x<- rgamma(n, shape=2, scale=4)
wilcox.test(y,x, alternative ="less")


use the alternative to specify less than or greater than depending on your setup. The default is not equal to as the alternative.

Mann-Whitney tests that it is equally likely that a randomly selected value from one sample will be less than or greater than a randomly selected value from a second sample.

If you have the measurements and not only the ranks and the measurements are numeric you could use a Kolmogorov-Smirnov test. ks.test(x,y) in R. To test ranks you could use the cor.test(x, y,alternative ="two.sided", method = "kendall") which tests that the two samples are correlated or not.

If your goal is to measure how close (equivalently how far away) are the two distributions from each other I would use the Kolmogorov-Smirnov test.