Aggregation of Correlations Coefficients (Spearman)

Question

in an analysis of survey data, I have to deal with multilevel/three-dimensional data. Now, I need to aggregate correlation coefficients found on the individual level (between individual rank-orders) and then compare these coefficients.

The original data looks like this: For each individual, 3 series á 15 items (agendas of public issues) have been measured. For a single individual, this could look like:

  i     A   B   C
  1     1   2   1
  2    15   7  12
  3     2  15   6
  4     3   1  11
  5     9   6   2
 ..        ...
 15     4   5   4

These are ranks, so I can compute the rank-correlations A~B and B~C on the individual level. Doing that, I result in a series of two correlations coefficients per case/individual.

CASE    rAB     rBC
   1   .213   .114
   2   .951   .524
   3  -.101   .022
   4   .607    1.0
 ...
 999   .549  -.661

Now I need to compare these coefficients to tell if A~B is larger then B~C (i.e., if the rank-order A is systematically more similar to B, than B is to C).

Of course, I could do simple t-tests over the two pairs. Yet, I doubt that correlation coefficients are scaled in such a way that adding/averaging them is allowed?

• I have read about Fisher's z-transformation for correlation coefficients, but in this data set, it it likely to have single cases with r=1 in the data - and a their z-value would be ∞ (Fisher's z-value is undefined for r=1), which makes averaging senseless.

• I could (not really) square the correlation coefficients to work with the explained variance r², but this would obviously conceil that some individual correlation are positive, while others are negative.

RQ: Are the agendas A and B more similar then the agendas B and C (systematically over 1000 individuals)? An agenda is a series of ranks or absolute values for each individual.

Update: Distribution of the correlations coefficients

Answer 1

I'm curious, since you have these ranks, why are you calculating correlation coefficients? I think a better approach would be to use a procedure that is designed to deal with ranks.

The Mann Whitney U test (or Wilcoxon Rank Sum test, they're the same thing) tests. This test concerns itself with whether or not the observations in group A are more likely to be greater than the observations in group B. In this test, the null hypothesis is the P(A > B) = P (B > A) (i.e. the observations from each population are equally likely to be larger than the other). If the test rejects, it's more likely that one group tends to have larger ranks than the other.

This seems to be exactly what your research question is, so I'd recommend this test instead of calculating correlation coefficients with these ranks. It's a fairly well known, commonly used test, so you shouldn't have any trouble justifying it to anyone either.

Let me know if you have any questions.

Answer 2

Mhhhh, so you would like to compare the distributions of correlation coefficients? How about to use a Kolmogorov-Smirnov test?

correpl <- t(replicate(1000, {
    scores <- rnorm(15)
    a <- rank(scores, ties.method = "r")
    ### experiment with sd = x to give some weights closer or farther to a
    b <- rank(scores + rnorm(15, sd = 5), ties.method = "r")
    c <- rank(scores + rnorm(15, sd = 6), ties.method = "r")
    return(c(cor(a,b), cor(b, c)))
  }))

c1 <- correpl[,1]
c2 <- correpl[,2]

curve(ecdf(c1)(x), from = -1, to = 1)
curve(ecdf(c2)(x), from = -1, to = 1, col = "red", add = T)
ks.test(c1, c2)

It is sensitive enough (too sensitive for my taste) to spot slightest differences and it is non-parametric. Try to experiment with the alternative setting.