I'm wondering what the best ways to compare (possibly ranked) lists when we know what the true ranking is and also the variable that decides the ranking.
Say this is the top 10 of a certain list, we know the ranking, and also the variable from which the ranking is obtained (Votes).
Player Votes 1 Ablett, Gary GC 28 528 Selwood, Joel GE 27 588 Swan, Dane CW 26 301 Johnson, Steve GE 25 120 Dangerfield, Patrick AD 22 236 Hannebery, Dan SY 21 464 Pendlebury, Scott CW 21 502 Rockliff, Tom BL 21 102 Cotchin, Trent RI 19 285 Jack, Kieren SY 19
Now, say I have two lists, produced by models A and B. The models have been trained on a independent set of data to predict the number of votes obtained by each player on a new dataset (from which the True Ranking is associated with).
Model A output:
1 Ablett, Gary GC 41 528 Selwood, Joel GE 30 588 Swan, Dane CW 29 211 Griffen, Ryan WB 28 464 Pendlebury, Scott CW 24 502 Rockliff, Tom BL 24 641 Watson, Jobe ES 23 301 Johnson, Steve GE 22 102 Cotchin, Trent RI 21 180 Fyfe, Nathan FR 21
Model B output:
1 Ablett, Gary GC 29.34127 588 Swan, Dane CW 25.49142 211 Griffen, Ryan WB 22.50983 464 Pendlebury, Scott CW 19.84517 528 Selwood, Joel GE 18.32023 120 Dangerfield, Patrick AD 16.94963 301 Johnson, Steve GE 16.05056 641 Watson, Jobe ES 15.73885 416 Montagna, Leigh SK 15.35478 339 Liberatore, Tom WB 14.50770
What are the best metrics or loss functions for determining whether model A or B's output is closer to the truth? I'm not sure if it is better to compare the rankings, or to compare the differences in votes for each player between lists. Are there optimal ways to do either? or does it depend on how one chooses to weight things?
In this setting, the position of the individuals in the list is probably more important than the number of votes obtained, but I imagine this information can still be used in some way. One of my concerns is that the number of ties in votes between individuals increases dramatically as you go down the 'True list'. In which case I imagine simply taking comparisons on the first 20 or 50 ranked entries might be helpful. Furthermore, the further down the list, the less important it is to be ranked correctly. e.g., It matters a great deal if the items in position 1 and 2 are swapped, but it's basically irrelevant if items 15 and 16 are swapped.