I'm wondering what the best ways to compare (possibly ranked) lists when we know what the true ranking is and also the variable that decides the ranking.

Say this is the top 10 of a certain list, we know the ranking, and also the variable from which the ranking is obtained (Votes).

True Ranking:

                     Player Votes
1           Ablett, Gary GC    28
528        Selwood, Joel GE    27
588           Swan, Dane CW    26
301       Johnson, Steve GE    25
120 Dangerfield, Patrick AD    22
236       Hannebery, Dan SY    21
464    Pendlebury, Scott CW    21
502        Rockliff, Tom BL    21
102       Cotchin, Trent RI    19
285         Jack, Kieren SY    19

Now, say I have two lists, produced by models A and B. The models have been trained on a independent set of data to predict the number of votes obtained by each player on a new dataset (from which the True Ranking is associated with).

Model A output:

1        Ablett, Gary GC    41
528     Selwood, Joel GE    30
588        Swan, Dane CW    29
211     Griffen, Ryan WB    28
464 Pendlebury, Scott CW    24
502     Rockliff, Tom BL    24
641      Watson, Jobe ES    23
301    Johnson, Steve GE    22
102    Cotchin, Trent RI    21
180      Fyfe, Nathan FR    21

Model B output:

1           Ablett, Gary GC 29.34127
588           Swan, Dane CW 25.49142
211        Griffen, Ryan WB 22.50983
464    Pendlebury, Scott CW 19.84517
528        Selwood, Joel GE 18.32023
120 Dangerfield, Patrick AD 16.94963
301       Johnson, Steve GE 16.05056
641         Watson, Jobe ES 15.73885
416      Montagna, Leigh SK 15.35478
339      Liberatore, Tom WB 14.50770

What are the best metrics or loss functions for determining whether model A or B's output is closer to the truth? I'm not sure if it is better to compare the rankings, or to compare the differences in votes for each player between lists. Are there optimal ways to do either? or does it depend on how one chooses to weight things?

In this setting, the position of the individuals in the list is probably more important than the number of votes obtained, but I imagine this information can still be used in some way. One of my concerns is that the number of ties in votes between individuals increases dramatically as you go down the 'True list'. In which case I imagine simply taking comparisons on the first 20 or 50 ranked entries might be helpful. Furthermore, the further down the list, the less important it is to be ranked correctly. e.g., It matters a great deal if the items in position 1 and 2 are swapped, but it's basically irrelevant if items 15 and 16 are swapped.

I am familiar with rank correlation methods, but are there other measures that might be more appropriate here? Are the measures mentioned here useful for solving this problem?



Your first issue appears to be that for some ranks, it is considered more important to be predicted correctly (usually the upper ranks), than others. So you should be looking into weighted rank correlation coefficients that can give to the top ranks' similarities/dissimilarities greater weight.

Here is some literature in case you are unfamiliar:

Pinto da Costa, J.F. and Soares, C. (2005). A weighted rank measure of correlation, Australian & New Zealand Journal of Statistics, 47(4), 515–529.

Authors' Abstract :Spearman's rank correlation coefficient is not entirely suitable for measuring the correlation between two rankings in some applications because it treats all ranks equally. In 2000, Blest proposed an alternative measure of correlation that gives more importance to higher ranks but has some drawbacks. This paper proposes a weighted rank measure of correlation that weights the distance between two ranks using a linear function of those ranks, giving more importance to higher ranks than lower ones. It analyses its distribution and provides a table of critical values to test whether a given value of the coefficient is significantly different from zero. The paper also summarizes a number of applications for which the new measure is more suitable than Spearman's.

The limit distribution of the above measure can be found in

JFP da Costa & LAC Roque (2006) : LIMIT DISTRIBUTION FOR THE WEIGHTED RANK CORRELATION COEFFICIENT $r_w$. REVSTAT – Statistical Journal Volume 4, Number 3, November 2006, 189–200

Another approach:

Maturi, T.A. and E.H. Abdelfattah, 2008. A New Weighted Rank Correlation. J. Math. Stat., 4: 226-230.

...and many more if you search "weighted rank correlation coefficient".

Your second issue, appears to be whether it is critical, important, or useful, to take into account the accuracy of the predicted votes rather than just the predicted rankings.

I have only a tentative thought here: prediction performance metrics, usually ignore whether the prediction under-predicts or over predicts, and consider absolute or squared deviations. In your case it appears useful to assess whether the two models tend to under-predict or over-predict. Perhaps you should examine their failures to predict the correct ranking, and see whether it was due to under-predicting or over-predicting the votes. I mean, assume that person $X$ with true rank $5$ was predicted as rank $6$ by model $A$. Was this because the votes of person $X$ were under-predicted? Or they were over-predicted but some other person's votes were also over-predicted even more? The vote-distances in the true data set appears as a possible normalizing factor here. This may lead to some conclusion regarding how "robust" the comparative evaluation of the two models is, when thinking of other data sets on which they may be applied. But I admit I am just throwing ideas around. I will try to maybe do a little theoretical search/work on this, and if I manage, I will update my answer.

| cite | improve this answer | |

The simple approach is simply to construct a loss function over rankings, like squared errors.

However since you are concerned about ties and would like to use voting data as well, you could try to model the cumulative distribution function (CDF) of the votes, which you could do either parametrically or non-parametrically. You then have 3 fitted CDFs: the truth, Model A and Model B. You can construct a distributional loss function based on the integrated sum of square differences between the distributions in votes.

You could then combine these two loss functions to construct a parametric weighted loss function of the two, such a $\alpha*L_{1}+(1-\alpha)*L_{2}$. You could then search over possible values of $\alpha$ that perform best.

| cite | improve this answer | |
  • $\begingroup$ Interesting approach. I'll try give it a crack shortly. $\endgroup$ – dcl Sep 10 '14 at 2:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.