# What is a good metric for my item ordering/ranking task?

My dataset is a collection of items with numerical features, and each of them has a score assigned to it as label.

My goal is to predict the ranking/order of the items, so the score prediction accuracy is less of concern, as long as the ranking by predicted socres are correct.

My question is, is there a good metric to evaluate my model on this task? I have checked out some learning to rank metrics, but they tend to emphasize the correct order of top items. However, in my case, it is equally important to put trailing items where they belong.

For example,

rank_metric([A, B, C, D, E], [A, B, C, D, E]) should be perfect rank_metric([A, B, C, D, E], [B, A, C, D, E]) should be slightly lower rank_metric([A, B, C, D, E], [A, B, D, C, E]) should be the same as the previous one rank_metric([A, B, C, D, E], [D, B, C, A, E]) should be punished even more

Supplementing earlier answer Keyword: Learning To Rank ( LTR) problem formulation Three prominent methods to solve ranking problems.

1. Pointwise ranking model
2. Pairwise ranking model
3. Listwise ranking model See https://en.wikipedia.org/wiki/Learning_to_rank & links therein. A very rich & matured field of machine learning.

Since you are not really interested in the correct score label for the members $$A, B, ...$$ of your dataset $$\Delta$$, but rather in the ranking, I suggest you do not train a model for the score but rather one for the ranking, by learning to order pairs.

Let $$s:\Delta\to\mathbb{R}$$ be the score. This defines a map:

$$r: \Delta^2 \to \{0, 1\}, \quad r(A, B): \left\{\begin{array}{ll} 0 & \mbox{, if } s(A) \le s(B)\\ 1 & \mbox{, otherwise} \end{array}\right.$$ You now learn this binary $$\Delta^2$$ classifier on your training data. Then, to find the ranking of some new set $$\Delta_{new}$$, you use $$\hat r$$ in an of-the-shelf sorting algorithm to order your set.

• Hi, thanks for the answer! Just following up, what models would you recommend to consume these feature pairs? Commented Mar 28, 2022 at 0:44
• In principle, any binary classifier would do. It mainly depends on your data and your constraints. E.g. if you need to understand how the classification is obtained from the input, i.e. if you are interested in the "why", you should use a model-based method with an easy model, e.g. logistic regression. If you are only interested in the result, you can also use non-parametric (or "near-nonparametric") methods like random forest, GBM, ... Commented Mar 28, 2022 at 4:47
• Thanks for the answer! Commented Mar 30, 2022 at 15:01
• @MarcZhang You're welcome. If you are satisfied with the answer, please consider accepting it. See: What should I do when someone answers my question? Commented Mar 30, 2022 at 15:22