2
$\begingroup$

I have a table with 15 items with attributes. The items are ranked, the rank is kind of the label. Now I build several algorithms, that get the 15 items in an un-ordered and tries to rank them. So far so good, but how can I evaluate/compare different rankings? Which one is better?

Example:

Original       Model 1 Results:   Model 2 Results:
Item | Rank    Item | Rank        Item | Rank
A    | 1       A    | 2           A    | 1
B    | 2       B    | 1           B    | 3
C    | 3       C    | 3           C    | 4
D    | 4       D    | 5           D    | 2
E    | 5       E    | 4           E    | 5

Which Model's ranking is closer to the original?

I already found Mean_reciprocal_rank, but it only seems to consider the best (rank 1) result.

What other options do I have to compare the rankings in order to evaluate and find my best ranking method?

Thanks!

PS: I'm pretty new to analytics, sorry if I'm not using correct terms or my questions seems to be stupid.

Cheers!

$\endgroup$
1
  • 1
    $\begingroup$ google this: Kendall’s Tau and Spearman’s Rank Correlation Coefficient $\endgroup$ – Eran Moshe Feb 15 '18 at 9:51
1
$\begingroup$

You can use something like the standard deviation, taking the square root of the average of the squared deviations of the ranking positions from their final value.

If the final ranking is a-b-c-d-e, you'll have

$\sigma_1 = \sqrt { {1+1+0+1+1} \over {5}}= 0.89 $ for b-a-c-e-d

$\sigma_2 = \sqrt { {0+4+1+1+0} \over {5}} = 1.10 $ for a-d-b-c-e

showing that the first model is better, even if it has four errors (while second model has three and correctly identify the first position).

$\endgroup$
3
  • $\begingroup$ I think you have an error for the second one since rank change for D is 2, not 4, and this way it is 0 + 2 + 1 + 1 + 0 which is the same as the first model. $\endgroup$ – divaka Aug 13 '19 at 19:40
  • $\begingroup$ Using your words, "rank change" for d is 2, so its squared deviation is 4, in fact. :) $\endgroup$ – Tomaso Neri Sep 15 '19 at 17:06
  • 1
    $\begingroup$ My bad, forgot about the square. Anyway, I like and appreciate your answers :) $\endgroup$ – divaka Sep 16 '19 at 19:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.