Can I compare ordinal rankings (and if so, how)? I have 15 supervisors who have ranked 21 employees from 1 (best) to 21 (worst), but the rankings are strictly ordinal-there is no interval data available.  Can I compare any two (or more) supervisors' rankings and determine how closely their evaluations are aligned? I'll be using Excel 2010 (or pen/paper) for my analysis, I don't have access to statistical software.
Sample Data (Is supervisor 3 more similar to 1 or 2)?
         Ranking    Supervisor 1    Supervisor 2    Supervisor 3
Employee 1          1               3               1
Employee 2          2               2               3
Employee 3          3               1               2

 A: I'm not sure how to do this in Excel, but Kendall's tau is what springs to mind. The gist of that method is for each pair of supervisors (1 and 2, say) to take each pair of employees and count how many times their ranks are ordered in the same way.
For that example, look at supervisor 1 and 3. They agree on the orderings of employees 1-2 and 1-3, but not on 2-3, so they'd have a tau of 1/3 (i.e. (2-1)/3).  Supervisors 1 and 2 disagree on all pairings, so they'd have a tau of -1 (i.e. (0-3)/3).
Addendum: You may also want to try Spearman's rho. It's more sensitive to outliers (for example, if one supervisor just hated someone and ranked them at the bottom, while the rest of the order was the same, they would have a low rho), so I don't think it's as good a measure, but it's trivially easy to calculate in Excel. Just do the CORREL of the ranks. The difference is a little like the a MAD vs RMSE difference; rho is more like the squared difference, while tau is more like the absolute difference.
A: Johann is recommending nonparametric measures of association.  If you reject independence than you are saying that the is a difference between the two groups.  Another way is nonparametric ANOVA which could use the Kruskal-Wallis test for differences between the three raters. Other association measures that were specifically designed for this problem are the Kappa statistic and the intraclass correlation.
