What is a good metric for my item ordering/ranking task?

Question

My dataset is a collection of items with numerical features, and each of them has a score assigned to it as label.

My goal is to predict the ranking/order of the items, so the score prediction accuracy is less of concern, as long as the ranking by predicted socres are correct.

My question is, is there a good metric to evaluate my model on this task? I have checked out some learning to rank metrics, but they tend to emphasize the correct order of top items. However, in my case, it is equally important to put trailing items where they belong.

For example,

rank_metric([A, B, C, D, E], [A, B, C, D, E]) should be perfect rank_metric([A, B, C, D, E], [B, A, C, D, E]) should be slightly lower rank_metric([A, B, C, D, E], [A, B, D, C, E]) should be the same as the previous one rank_metric([A, B, C, D, E], [D, B, C, A, E]) should be punished even more

Thanks in advance!

Answer 1

Supplementing earlier answer Keyword: Learning To Rank ( LTR) problem formulation Three prominent methods to solve ranking problems.

1. Pointwise ranking model
2. Pairwise ranking model
3. Listwise ranking model See https://en.wikipedia.org/wiki/Learning_to_rank & links therein. A very rich & matured field of machine learning.

Answer 2

Since you are not really interested in the correct score label for the members $A, B, ...$ of your dataset $\Delta$, but rather in the ranking, I suggest you do not train a model for the score but rather one for the ranking, by learning to order pairs.

Let $s:\Delta\to\mathbb{R}$ be the score. This defines a map:

$$ r: \Delta^2 \to \{0, 1\}, \quad r(A, B): \left\{\begin{array}{ll} 0 & \mbox{, if } s(A) \le s(B)\\ 1 & \mbox{, otherwise} \end{array}\right. $$ You now learn this binary $\Delta^2$ classifier on your training data. Then, to find the ranking of some new set $\Delta_{new}$, you use $\hat r$ in an of-the-shelf sorting algorithm to order your set.