Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 4 down vote favorite

I have 15 supervisors who have ranked 21 employees from 1 (best) to 21 (worst), but the rankings are strictly ordinal-there is no interval data available. Can I compare any two (or more) supervisors' rankings and determine how closely their evaluations are aligned? I'll be using Excel 2010 (or pen/paper) for my analysis, I don't have access to statistical software.

Sample Data (Is supervisor 3 more similar to 1 or 2)?

         Ranking    Supervisor 1    Supervisor 2    Supervisor 3
Employee 1          1               3               1
Employee 2          2               2               3
Employee 3          3               1               2
share|improve this question
After I posted, I noticed that what was supposed to be a table of sample data is simply a mushed up line of nothing. I looked for help formating the table, but couldn't find anything. So, if anyone can point me in the right direction, I would appreciate it and will clean the mess up. – David Vandenbos May 31 '12 at 12:55
Is that roughly what the table is supposed to look like? – Macro May 31 '12 at 13:01
Thanks Macro, that's perfect. How did you enter the table? – David Vandenbos May 31 '12 at 14:42
1  
No problem. If you indent with four spaces, it turns it into "computer text". At that point, I just spaced it until it looked right. – Macro May 31 '12 at 14:44

2 Answers

up vote 4 down vote accepted

I'm not sure how to do this in Excel, but Kendall's tau is what springs to mind. The gist of that method is for each pair of supervisors (1 and 2, say) to take each pair of employees and count how many times their ranks are ordered in the same way.

For that example, look at supervisor 1 and 3. They agree on the orderings of employees 1-2 and 1-3, but not on 2-3, so they'd have a tau of 1/3 (i.e. (2-1)/3). Supervisors 1 and 2 disagree on all pairings, so they'd have a tau of -1 (i.e. (0-3)/3).

Addendum: You may also want to try Spearman's rho. It's more sensitive to outliers (for example, if one supervisor just hated someone and ranked them at the bottom, while the rest of the order was the same, they would have a low rho), so I don't think it's as good a measure, but it's trivially easy to calculate in Excel. Just do the CORREL of the ranks. The difference is a little like the a MAD vs RMSE difference; rho is more like the squared difference, while tau is more like the absolute difference.

share|improve this answer
Thanks for the help. For expedience, I ended up using Excel's correlation function and that got seems to have gotten what I need. I found some VBA to implement Kendall's Tau in Excel and will experiment more with that later. Also, tried R again and remembered why I usually stick with Excel. – David Vandenbos Jun 1 '12 at 13:42
up vote 1 down vote

Johann is recommending nonparametric measures of association. If you reject independence than you are saying that the is a difference between the two groups. Another way is nonparametric ANOVA which could use the Kruskal-Wallis test for differences between the three raters. Other association measures that were specifically designed for this problem are the Kappa statistic and the intraclass correlation.

share|improve this answer
You mention "if I reject independence". Quite frankly I'd never even considered indpendence. Can you elaborate a bit more on what that means, whether its a good idea (or not) and how I can tell the difference? – David Vandenbos May 31 '12 at 14:50
1  
Really in this case independence is the same as no difference in the proportions. That would involve look at your data as a contingency table with the columns being superv. 1 vs 2, suprev. 1 vs 3 and superv. 2 vs 3 and the rows being agree or disagree giving a 2x3 contingency table. You define agreement. It may be that they gave exactly the same score or that the scores were close in some well-defined sense. Each cell is a count of the number of times a particular pair of supervisors agree (or disagree). – Michael Chernick May 31 '12 at 15:34
1  
Independence then becomes the probability of a specific joint probability being equal to the product of two individual probabilities. It is mathematically equavalent to saying that the proportion of agreement for each pair of supervisors is the same as for any other pair. – Michael Chernick May 31 '12 at 15:36
Ok, that makes sense. Thanks for the extra info. I'll have to think about how much that applies to my case. – David Vandenbos May 31 '12 at 16:34
You don't have to formulate your problem that way. I think the Kruskal-Wallis test is very straight forward and applicable. I mentioned independence in the case of contingency tables because that is what measures like Kendall's tau do. They look at concordance and discordance to se if either gets particularly high. – Michael Chernick May 31 '12 at 16:39

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.