# Covariance matrix for ranking data

I am testing concordance between boys and girls on their rankings of 9 items. The literature says this can be done by comparing their mean rank vectors under the null hypothesis that the mean rank are equal. The test statistics requires the covariance matrix S, which is an 8x8 dim estimated covariance matrix of the rank vectors of the 2 groups. Any idea how I estimate S, given the mean rank vector of the boys' and the girls' groups? Thanks.

• A few Q's: What is the literature you're referring to? Is there a specific test you're trying to implement? Also, why would the covariance matrix be 8x8? This sounds vaguely like the test devised by Hollander & Sethuraman (1978) but I could be wrong – awhug Mar 25 at 10:37
• It is Yu's survey article, and yes, this is a permutation test for which as you identified he references Hollander + Sethuraman . It is 8x8 dimensional since the rank of the last item is fully determined based on the other 8 items. The test statistic is (n1xn2)/(n1+n2) *(r1bar - r2bar)'S^-1(r1bar-r2bar). Here the rbar vectors are the mean rank vectors of the 2 groups being compared. Yu says that S is the estimated covariance matrix of the rank vectors from the two groups under H0=Mean rank vectors are the same for both groups. How do I compute S? – user2450223 Mar 25 at 12:10

In this case it's just a regular covariance matrix on all the observed rankings (i.e. across both groups) with one of the items omitted, following Hollander and Sethuraman (1978). In R you could calculate this using the cov function.

For example, you can quite easily reproduce the example analysis in Hollander & Sethuraman (1978). The data is a set of rankings from retiree women on who they'd rather spend their leisure time with (men, women, or both), by race (white, black), collected by Cindy Sutton.

# Input the data
leisure <- data.frame(
men = c(1, 1, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3),
women = c(2, 3, 3, 3, 3, 3, 3, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2),
both = c(3, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 3, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1),
group = c(rep("black", 13), rep("white", 14)))

# Get group sample sizes; M = number of items - 1
n <- with(leisure, table(group))
k <- 3 - 1

# Calculate average ranks by group omitting last item, find differences
avg_group_1 <- colMeans(subset(leisure, group == "white", select = 1:k))
avg_group_2 <- colMeans(subset(leisure, group == "black", select = 1:k))
avg_diffs <- avg_group_1 - avg_group_2

# Calculate covariance matrix on all ranks, omitting last item
S <- cov(leisure[1:k])

# Calculate test statistic
chi_sq <- (prod(n) / sum(n)) %*% t(avg_diffs) %*% solve(S) %*% avg_diffs
p_val <- 1 - pchisq(chi_sq, k - 1)

# Print result - Same as H&S (1978), p. 17
paste0("Chi-Square Test Statistic = ", round(chi_sq, 3), ", p = ", round(p_val, 4))
# "Chi-Square Test Statistic = 13.849, p = 2e-04"


Actually, with a little rescaling this test statistic is equivalent to a Pillai trace, as shown by Beasley (2000). In other words, you could just as well run a MANOVA (again omitting one item) on the ranking data to find the same result.

# MANOVA with the Pillai test
summary(manova(cbind(males, females) ~ group, leisure), test = "Pillai") # = 0.53267
0.53267 * (sum(n) - 1) # Same result


For what it's worth, I've not often seen too much use for this older test. I'd usually be more interested in which items the two groups differ on specifically. To that end, I'd probably prefer a rank-ordered logit or multinomial probit model, but that's just me.

### References

Beasley, T. M. (2000). Nonparametric tests for analyzing interactions among intra-block ranks in multiple group repeated measures designs. Journal of Educational and Behavioral Statistics, 25(1), 20-59.

Hollander, M., & Sethuraman, J. (1978). Testing for agreement between two groups of judges. Biometrika, 65(2), 403-410.

• Thanks, this is a really great answer. Appreciate the references! – user2450223 Mar 26 at 11:04