0
$\begingroup$

I have two result sets of identical search queries run under different parameters, analysis, etc. I will refer to them as RS1 and RS2. RS1 and RS2 may not have the same length, but they are rank-ordered with the first entry in the set being the highest priority. If RS1 is the "desired" set, or perhaps "control set", what statistical measures can I employ to determine how far off RS2 is from RS1?

For example, if the result sets return user ids:

RS1: [2, 4, 6], RS2: [4, 6, 2]

We can see that RS1 and RS2 are decently similar by inspection.

But in this example:

RS1: [2, 3, 4], RS2: [2, 6, 7]

is a much "worse" match. Obviously preferences of how important order is, etc matter, but what will be a good starting point? Any resources I could check out?

$\endgroup$
1
  • $\begingroup$ I have heard of Jaccard index, but I think the measure needs to be more complicated than that since rank matters. $\endgroup$ Jun 20, 2017 at 21:13

1 Answer 1

1
$\begingroup$

You can compare the medians of the two distributions via Mann-Whitney test.

The test in R is implemented in the wilcox.test function in base R.

The R code to do perform both of these tests is:

n<-100
y<- rgamma(n, shape=1, scale=1)
x<- rgamma(n, shape=2, scale=4)
wilcox.test(y,x, alternative ="less")

use the alternative to specify less than or greater than depending on your setup. The default is not equal to as the alternative.

Mann-Whitney tests that it is equally likely that a randomly selected value from one sample will be less than or greater than a randomly selected value from a second sample.

If you have the measurements and not only the ranks and the measurements are numeric you could use a Kolmogorov-Smirnov test. ks.test(x,y) in R. To test ranks you could use the cor.test(x, y,alternative ="two.sided", method = "kendall") which tests that the two samples are correlated or not.

If your goal is to measure how close (equivalently how far away) are the two distributions from each other I would use the Kolmogorov-Smirnov test.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.