in an analysis of survey data, I have to deal with multilevel/three-dimensional data. Now, I need to aggregate correlation coefficients found on the individual level (between individual rank-orders) and then compare these coefficients.

The original data looks like this: For each individual, 3 series á 15 items (agendas of public issues) have been measured. For a single individual, this could look like:

  i     A   B   C
  1     1   2   1
  2    15   7  12
  3     2  15   6
  4     3   1  11
  5     9   6   2
 ..        ...
 15     4   5   4

These are ranks, so I can compute the rank-correlations A~B and B~C on the individual level. Doing that, I result in a series of two correlations coefficients per case/individual.

CASE    rAB     rBC
   1   .213   .114
   2   .951   .524
   3  -.101   .022
   4   .607    1.0
 999   .549  -.661

Now I need to compare these coefficients to tell if A~B is larger then B~C (i.e., if the rank-order A is systematically more similar to B, than B is to C).

Of course, I could do simple t-tests over the two pairs. Yet, I doubt that correlation coefficients are scaled in such a way that adding/averaging them is allowed?

  • I have read about Fisher's z-transformation for correlation coefficients, but in this data set, it it likely to have single cases with r=1 in the data - and a their z-value would be ∞ (Fisher's z-value is undefined for r=1), which makes averaging senseless.

  • I could (not really) square the correlation coefficients to work with the explained variance r², but this would obviously conceil that some individual correlation are positive, while others are negative.

RQ: Are the agendas A and B more similar then the agendas B and C (systematically over 1000 individuals)? An agenda is a series of ranks or absolute values for each individual.

Update: Distribution of the correlations coefficients

Distribution of correlation coefficients with normal distribution

  • 2
    $\begingroup$ We use z-transform when comparing Pearson correlations from 2 independent samples. I.e. two statistics are compared. In your second table, a correlation is just some data value, not statistic. We actually don't know in advance what distribution under null these values follow [and that is in particular because your 15 items were not random observations]; we might assume it is normal, for example. Then paired t-test between two columns would be reasonable. $\endgroup$
    – ttnphns
    Jul 14, 2016 at 10:43
  • $\begingroup$ @ttnphns: Very good point, thank you! Actually, the coefficients in my data nearly follow a normal distribution. And now you wrote that, I also remember some rule that their distribution is not soo important, if you have a large number of cases... It really makes me happy, when things turn out to be easy. Would you post your comment as response, so I can accept the answer? $\endgroup$
    – BurninLeo
    Jul 14, 2016 at 20:10
  • $\begingroup$ You can check this thread for example of Fisher z-transform used with correlation coefficients: stats.stackexchange.com/questions/221246/… $\endgroup$
    – Tim
    Jul 20, 2016 at 8:32
  • $\begingroup$ Thanks. As noted in the question, a Fisher z-transformation is not applicable in this case, because there are coefficients with r=+1 or -1. $\endgroup$
    – BurninLeo
    Jul 20, 2016 at 9:17

2 Answers 2


I'm curious, since you have these ranks, why are you calculating correlation coefficients? I think a better approach would be to use a procedure that is designed to deal with ranks.

The Mann Whitney U test (or Wilcoxon Rank Sum test, they're the same thing) tests. This test concerns itself with whether or not the observations in group A are more likely to be greater than the observations in group B. In this test, the null hypothesis is the P(A > B) = P (B > A) (i.e. the observations from each population are equally likely to be larger than the other). If the test rejects, it's more likely that one group tends to have larger ranks than the other.

This seems to be exactly what your research question is, so I'd recommend this test instead of calculating correlation coefficients with these ranks. It's a fairly well known, commonly used test, so you shouldn't have any trouble justifying it to anyone either.

Let me know if you have any questions.

  • $\begingroup$ Thank for your answer. The (rank-)correlation coefficients were calculated to have a measure of similarity on the individual level. Please note that A, B, and C are 15 values per individual (overall this makes 3-dimensional data/multilevel-data). If I am correct, the U-test only works on "traditional" series of data (values x cases)? $\endgroup$
    – BurninLeo
    Jul 14, 2016 at 19:45

Mhhhh, so you would like to compare the distributions of correlation coefficients? How about to use a Kolmogorov-Smirnov test?

correpl <- t(replicate(1000, {
    scores <- rnorm(15)
    a <- rank(scores, ties.method = "r")
    ### experiment with sd = x to give some weights closer or farther to a
    b <- rank(scores + rnorm(15, sd = 5), ties.method = "r")
    c <- rank(scores + rnorm(15, sd = 6), ties.method = "r")
    return(c(cor(a,b), cor(b, c)))

c1 <- correpl[,1]
c2 <- correpl[,2]

curve(ecdf(c1)(x), from = -1, to = 1)
curve(ecdf(c2)(x), from = -1, to = 1, col = "red", add = T)
ks.test(c1, c2)

It is sensitive enough (too sensitive for my taste) to spot slightest differences and it is non-parametric. Try to experiment with the alternative setting.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.