The problem is predicting the ranking of a list using some features e.g.
rank_metric([A, B, C, D, E], [A, B, C, D, E]) should be 1
rank_metric([A, B, C, D, E], [B, A, C, D, E]) should be lower
rank_metric([A, B, C, D, E], [E, A, B, C, D]) should be even lower
I came across Kendall's τ while looking for a good metric to use. Is this a recommended metric for this case and are there others that are more suitable?
Also if I wanna measure the performance based on only the first predicted item what would be a good measure then? a simple solution would be to just have something like
1 - actual_index_of_predicted/len(list)