What is a good metric to evaluate ranking without retrieval?

Question

The problem is predicting the ranking of a list using some features e.g.
rank_metric([A, B, C, D, E], [A, B, C, D, E]) should be 1
rank_metric([A, B, C, D, E], [B, A, C, D, E]) should be lower
rank_metric([A, B, C, D, E], [E, A, B, C, D]) should be even lower
I came across Kendall's τ while looking for a good metric to use. Is this a recommended metric for this case and are there others that are more suitable?

Also if I wanna measure the performance based on only the first predicted item what would be a good measure then? a simple solution would be to just have something like
1 - actual_index_of_predicted/len(list)

Answer 1

You may also want to check the normalized version of Damerau–Levenshtein distance. It is technically a string metric for measuring the edit distance between two sequences. However, in this case, it can be used as a ranking metric. A python package called pyxDamerauLevenshtein implements this distance metric.

Answer 2

You can compute the probability that a randomly chosen pair of elements from your list is ranked correctly. This generalizes ROC AUC, which is equivalent to the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative instance for binary classification. The advantage of this metric is swapping any two successively ranked elements changes performance by a fixed amount. So in your example,

rank_metric([A, B, C, D, E], [A, B, C, D, E]) is 1

rank_metric([A, B, C, D, E], [B, A, C, D, E]) is $1 - \frac{1}{10}$

rank_metric([A, B, C, D, E], [B, C, A, D, E]) is $1 - \frac{2}{10}$

rank_metric([A, B, C, D, E], [B, C, A, D, E]) is $1 - \frac{2}{10}$

rank_metric([A, B, C, D, E], [E, D, C, B, A]) is $1 - \frac{10}{10}=0$

You can find an N log N implementation of this metric here.