I have some process A that assigns values y_i for i=1:N to instances x_i. N may be very high, on the order of millions. If I have some process B that assigns values y_i' to those same instances, then I would like to test whether the ranking produced by B is statistically similar to the one produced by A.

The Wilcoxon-signed rank test won't do here because it's the opposite of what I'm trying to show. Wilcoxon's null hypothesis is that the two rankings are equal. On the other hand, I am trying to reject a null hypothesis of significant difference in ranking.

I would prefer something that doesn't assume a normal distribution of values and that can be algebraically reasoned about, since what I am ultimately trying to show is that a certain algorithm can do a good job of approximating a true ranking.

  • $\begingroup$ Perhaps something like equivalence testing (TOST) is nearer to what you want. $\endgroup$ – Glen_b Jun 9 '15 at 12:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.