3
$\begingroup$

I have $N$ objects $A_1,\dots,A_N$. I have two algorithms that assign two different scores to each of this objects. Thus sorting on this scores gives me two different rankings of these same objects. Now how do I measure the similarity (or dissimilarity) of these different rankings in a visually understandable manner. Are their known approaches to this?

$\endgroup$
  • $\begingroup$ A Spearman correlation would capture the ordinality in the agreement of the rankings. Visualizing that relationship could be as simple as creating deciles for each set of scores and cross-classifying them (i.e., a crosstab). Next, by highlighting with different colors the agreement on the diagonal versus the diagonal plus one or two deciles off and summarizing the percentages of each with a few numbers should communicate what you need. $\endgroup$ – Mike Hunter Oct 16 '15 at 13:20
  • 1
    $\begingroup$ What is the purpose of the comparison? Are these rankings according to different criteria, or are they supposed to use the same criterion, but made by different judges? ... Probably these matters. $\endgroup$ – kjetil b halvorsen Oct 16 '15 at 13:20
  • $\begingroup$ These rankings are according to different criterion, but there should be certain natural things I expect in them. For instance, certain objects are really good and I naturally expect to them to score higher. However Algorithm 1's criterion doesn't reflect this whereas Algorithm 2 does. I do have a rigorous explanation of this effect. However, I don't have a nice way of putting the numbers. $\endgroup$ – dineshdileep Oct 16 '15 at 13:44
1
$\begingroup$

I came up with a solution that could be called a "combined rank plot". The idea is that if each ranking is sorted in ascending order, then the difference of an index from the first position can be used a type of "rank loss function", and adding these "loss" values gives a "combined rank" loss measure. I.e., the top ranking item has zero loss, the second best rank has a loss of one, etc. Then, these individual loss values are added to compute the overall loss function value for each item with respect to the two separate rankings. This "combined rank loss" function can be used to visualize the relationship between the two ranks. There are more than one way to achieve the same combined rank loss from two given rankings for each item. Therefore, I designed the plot to position the combined rank centrally, between two parallel axes: one for each of the original rankings. Because the highest ranking naturally corresponds to the highest position on a list, the parallel axes are pointing down, so that the lowest rank losses appear at the top of the plot (similar to a sorted list with the best choices on top). Attached is an image illustrating this plot with randomized inputs as an example (using Matlab). I am happy to share the code if neededcombined rank plot.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.