3
$\begingroup$

Let's say I have 5 teams: "a", "b", "c", "d", "e". As we go through the season, I take snapshots of how they rank. I then want to find the earliest point at which the ranking is a "good" predictor of the ranking at the end of the season. How can I measure and compare the accuracy (similarity to the final ranking) of these earlier rankings?

The first think I thought of was to just look at the number of slots that were guessed correctly, but then I can end up at a place where "e"-"a"-"b"-"c"-"d" will count as less accurate that "b"-"e"-d"-"c"-"a" since the latter is at least 20% correct. This feels like the type of situation common enough to have a standard approach, sadly I haven't been able to find one.

$\endgroup$
1
$\begingroup$

You can use something like the standard deviation, taking the square root of the average of the squared deviations of the ranking positions from their final value.

If the final ranking is a-b-c-d-e, you'll have

$\sigma_1 = \sqrt { {1+1+1+1+16} \over {5}}= 2.00 $ for e-a-b-c-d

$\sigma_2 = \sqrt { {1+9+1+1+16} \over {5}} = 2.37 $ for e-a-b-c-d

showing that the first ranking is better, and a-c-b-d-e is nearly perfect ($\sigma_3 = 0.63$) even if it has two errors like e-b-c-d-a (that's the worst, having $\sigma_4 = 2.52$).

You can refine and enhance this approach if your ranking comes from a ranking system, instead of from a simple points counting ranking.

$\endgroup$
  • $\begingroup$ Thanks! I think this is a more scaleable solution for cases when the number of elements ranked are high, and works well to quantify how far from reality you are with your predictions. $\endgroup$ – NeonBlueHair Oct 28 '16 at 21:30
0
$\begingroup$

Another way to formulate the ranking problem is to say you have 10 pairs (5*(5-1)/2), and want to put those in the right order. You could then compare rankings by comparing how many pairs are in the right order. So the first ordering contains the pairs:
ea, eb, ec, ed, ab, ac, ad, bc, bd, cd
Of these 60% are in the right order (namely, the last 6 pairs). On the other hand for the second ordering we have:
be, bd, bc, ba, ed, ec, ea, dc, da, ca
Here only 30% of the pairs are in the right order. Of course the number of pairs goes up quadratically as the number of elements to be ordered increases. So that is something to keep in mind. Also, in applications like a search engine, you might only be interested in the accuracy of the top results. In that case, different comparison functions that focus on those top results are better.

$\endgroup$
  • $\begingroup$ I like this idea, but like you said it does become difficult as the number of elements being ranked increases. $\endgroup$ – NeonBlueHair Oct 28 '16 at 21:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.