Weighted Kendall tau rank correlation coefficient

Question

I need to use a measure to compare two rankings:

Consider the ground truth as 1,2,3,4,5,6. Let two rankings be R1 and R2.

R1: 1,2,3,6,4,5
R2: 1,4,2,3,5,6

From Wikipedia Kendall's tau ranking correlation coefficient is given by: $$ \frac{\text{Number of Concordant Pairs - Number of Discordant Pairs}}{(n(n-1)/2) } $$ Calculation of Kendall's tau for R1:

Number of Concordant Pairs = 15
Number of Discordant Pairs = 2

Similarly for R2:

Number of Concordant Pairs = 15
Number of Discordant Pairs = 2

Thus, although the Kendall's correlation coefficient is same for both the lists, I want to penalise R2 more as the differences in position is towards the head than tail. Thus, along with the ranking, I also want to take into consideration the position. Is there a well defined measure for this? How can this be done?

Answer 1

I don't have commenting privileges, so I will attempt an answer here. Perhaps your original question is unclear, but here are answers depending on your exact meaning:

"I want to penalise R2 more as the differences in position is towards the head than tail. Thus, along with the ranking, I also want to take into consideration the position."

• If you want to penalize R2 because it moved too far towards position 1, then despite your response to another answer, you do care about relevance. In other words, if errors too far towards the head or towards the tail matter, than relevance-based ranking is what you are looking for. The other answer's suggestion of Discounted cumulative gain is a good choice.
• Alternatively, I don't know if you think that in R2 that there was a bigger absolute change or jump in the ranking, for which you want to give a penalty. In fact, the difference in both cases is -2: In R1, 6 moved -2 to rank 4; in R2, 4 moved -2 to 2. Thus, Kendall's tau is identical, because tau only cares about how much difference there is, not where exactly the jump occurred. If for instance, there had only been a jump in 1 (e.g. if R3 were to be 2,1,3,4,5,6), then tau would have a larger value (indicating more concordance). If that's the case, then Kendall's tau might be just what you need.

Answer 2

A positional weighted kendall-tau (a.k.a Kemeny) metric would do the work. This is a generalization of the original metric with assigning weights on possible swaps on consecutive positions (hence an infinite class).

A simple example would be assigning weights to a swap between 1st and 2nd alternatives, 2nd and 3rd, so on and so forth.

You can see more here:

http://www.sciencedirect.com/science/article/pii/S0304406814000068

Or here an extended free access version: http://digitalarchive.maastrichtuniversity.nl/fedora/get/guid:d5f76b52-4b10-4123-9e5f-f50f53067abc/ASSET1

Answer 3

I don't know if it is possible with Kendall tau, but some ranking measures such as Discounted cumulative gain naturally penalize more inversions towards some extreme of the list.