I have $N$ objects $A_1,\dots,A_N$. I have two algorithms that assign two different scores to each of this objects. Thus sorting on this scores gives me two different rankings of these same objects. Now how do I measure the similarity (or dissimilarity) of these different rankings in a visually understandable manner. Are their known approaches to this?
I came up with a solution that could be called a "combined rank plot". The idea is that if each ranking is sorted in ascending order, then the difference of an index from the first position can be used a type of "rank loss function", and adding these "loss" values gives a "combined rank" loss measure. I.e., the top ranking item has zero loss, the second best rank has a loss of one, etc. Then, these individual loss values are added to compute the overall loss function value for each item with respect to the two separate rankings. This "combined rank loss" function can be used to visualize the relationship between the two ranks. There are more than one way to achieve the same combined rank loss from two given rankings for each item. Therefore, I designed the plot to position the combined rank centrally, between two parallel axes: one for each of the original rankings. Because the highest ranking naturally corresponds to the highest position on a list, the parallel axes are pointing down, so that the lowest rank losses appear at the top of the plot (similar to a sorted list with the best choices on top). Attached is an image illustrating this plot with randomized inputs as an example (using Matlab). I am happy to share the code if needed.