Is there a test to measure correlation between two sets of data that are based on rank (more details provided)

Question

I'm trying to design an experiment to validate simulation software that I'm using. For talking purposes, let's say I'm testing seven different designs, A, B, C, D, E, F, and G. The seven different designs will be ranked according to a particular performance parameter that will be tested for both in real life and using the simulation software. For example, let's say the goal is to maximize velocity. Real life testing shows the designs rank C, A, G, E, D, F, and B from fastest to slowest. If the simulation software also shows that the fastest to slowest designs are C, A, G, E, D, F and B then there is perfect correlation and the software would be appear to be useful. However, if the software ranked the designs A, C, G, E, D, F, and B, the correlation is no longer perfect. And, if the software ranked the designs B, A, G, E, D, F, and C, the correlation is even further from perfect because instead of only the first and second place designs being switched in ranking, now the first and last place designs are switched. The importance of this is that even though the software got 5 out of 7 ranks correct in both cases, the error in the second case is greater than the first since the first and last ranks are switched instead of just the first two. I would like a statistical test that quantifies such correlations, taking into account not only the number of ranks correctly identified, but also how far off the incorrect ranks are from being correct.

Any help is greatly appreciated, thanks.

Answer 1

There are several measures of rank correlation.

The two most common ones are:

(i) Kendall's tau. This is based on counting the proportion of times both sets of ranks move to a higher rank together or a lower rank together (move in the same direction in the ordering) when considering all pairs of corresponding ranks $(x_i,y_i)$ vs $(x_j,y_j)$.

(ii) Spearman's rho. This is simply the ordinary correlation coefficient (Pearson correlation) computed on the ranks.

Either set of ranking will give you a partial order on which you can say things like "these two rankings are more alike than these two".

Wikipedia:
Rank correlation
Spearman's rho
Kendall's tau

(Knowing whether that solves whatever your underlying problem is would require understanding more about what you're trying to achieve there.)

You might also want to look into measures of inter-rater agreement.

Wikipedia:
inter-rater agreement

---

On your examples:

1. (C, A, G, E, D, F, B) vs (C, A, G, E, D, F, B)

   then both measures I mentioned would give correlations of "1".

2. (C, A, G, E, D, F, B) vs (A, C, G, E, D, F, B)

   Kendall: 0.905 Spearman: 0.964

3. (C, A, G, E, D, F, B) vs (B, A, G, E, D, F, C)

   Kendall: -0.047 Spearman: -0.285

(Kendall correlations tend typically to be smaller in magnitude but the relative comparisons are usually similar)

Here's what the rankings of the designs are:

  design orda ordb1 ordb2
1      A    2     1     2
2      B    7     7     1
3      C    1     2     7
4      D    5     5     5
5      E    4     4     4
6      F    6     6     6
7      G    3     3     3

Measuring correlation in R:

> a
[1] "C" "A" "G" "E" "D" "F" "B"
> b1
[1] "A" "C" "G" "E" "D" "F" "B"
> b2
[1] "B" "A" "G" "E" "D" "F" "C"

cor(order(a),order(b1),method="kendall")
cor(order(a),order(b1),method="spearman")

cor(order(a),order(b2),method="kendall")
cor(order(a),order(b2),method="spearman")