# Can I compare ordinal rankings (and if so, how)?

I have 15 supervisors who have ranked 21 employees from 1 (best) to 21 (worst), but the rankings are strictly ordinal-there is no interval data available. Can I compare any two (or more) supervisors' rankings and determine how closely their evaluations are aligned? I'll be using Excel 2010 (or pen/paper) for my analysis, I don't have access to statistical software.

Sample Data (Is supervisor 3 more similar to 1 or 2)?

         Ranking    Supervisor 1    Supervisor 2    Supervisor 3
Employee 1          1               3               1
Employee 2          2               2               3
Employee 3          3               1               2

-
After I posted, I noticed that what was supposed to be a table of sample data is simply a mushed up line of nothing. I looked for help formating the table, but couldn't find anything. So, if anyone can point me in the right direction, I would appreciate it and will clean the mess up. –  dav May 31 '12 at 12:55
Is that roughly what the table is supposed to look like? –  Macro May 31 '12 at 13:01
Thanks Macro, that's perfect. How did you enter the table? –  dav May 31 '12 at 14:42
No problem. If you indent with four spaces, it turns it into "computer text". At that point, I just spaced it until it looked right. –  Macro May 31 '12 at 14:44

I'm not sure how to do this in Excel, but Kendall's tau is what springs to mind. The gist of that method is for each pair of supervisors (1 and 2, say) to take each pair of employees and count how many times their ranks are ordered in the same way.

For that example, look at supervisor 1 and 3. They agree on the orderings of employees 1-2 and 1-3, but not on 2-3, so they'd have a tau of 1/3 (i.e. (2-1)/3). Supervisors 1 and 2 disagree on all pairings, so they'd have a tau of -1 (i.e. (0-3)/3).

Addendum: You may also want to try Spearman's rho. It's more sensitive to outliers (for example, if one supervisor just hated someone and ranked them at the bottom, while the rest of the order was the same, they would have a low rho), so I don't think it's as good a measure, but it's trivially easy to calculate in Excel. Just do the CORREL of the ranks. The difference is a little like the a MAD vs RMSE difference; rho is more like the squared difference, while tau is more like the absolute difference.

-
Thanks for the help. For expedience, I ended up using Excel's correlation function and that got seems to have gotten what I need. I found some VBA to implement Kendall's Tau in Excel and will experiment more with that later. Also, tried R again and remembered why I usually stick with Excel. –  dav Jun 1 '12 at 13:42