# Significance testing between two mean correlation values

I have collected 50 data points from 9 individuals for 4 variables, say, a,b,c,d. For each individual, I correlated ab and cd using a Pearson correlation.

Aim: I would like to test whether across all 9 individuals, correlation(ab) is significantly different to correlation(cd).

Thus, I took an average of all the correlation(ab) values after Fisher r-to-z transforming them, and did the same for correlation(cd).

I would now like to test whether or not these mean correlation values are significantly different. I have found this online calculator: http://vassarstats.net/rdiff.html

In this calculator, I will set r equal to my mean correlations. I am wondering if n should be set to the total number of data points across all 9 individuals (450), or if n should equal the number of individuals the mean correlation was determined from (9)?

• There are better ways to analyze this data than a comparison of Pearson correlations. Have you considered other methodological options? E.g., ANOVA? Commented Oct 19, 2016 at 21:36
• Thank you all for your suggestions! I will try fitting a linear model to these data and will try comparing R-squared values using ANOVA. Does this sound sensible? I read this link Commented Oct 19, 2016 at 22:13
• Why not do it using contrasts? en.wikipedia.org/wiki/Contrast_(statistics) Commented Oct 19, 2016 at 23:45
• Does anyone share my concern about using 50 observations from 9 individuals? I would think that the individual should be a random effect.
– Dave
Commented Jun 7, 2019 at 13:57

Short answer: put n=426 in the calculator, but beware of assumptions detailed in long answer.

I assume here that your null hypothesis is that correlation between a and b is the same than correlation between c and d.

The first thing to do would be to test for correlation between a-b correlation and c-d correlation. If they are uncorrelated maths became a lot simpler, although if you can estimate correlation between both correlation r-z transform could still be useful. From here I will assume that a-b correlation is not correlated to c-d correlation.

Assuming that a and b are close enough to a normal distribution, every 50 sample a-b correlation, once transformed, will result in a value distributed normally with mean ${1 \over 2}\ln\left({{1+\rho} \over {1-\rho}}\right)$ and variance $\frac{1}{N-3}$ (with $N=50$). Under the null hypothesis that a-b and c-d have the same correlation, transformation of c-d correlations will also be normally distributed with the same mean and variance.

The average of correlations of 9 samples will be normally distributed with the same mean and $1/9$ of the variance. That is:

$$var(mean(correlation(a,b)))=var(mean(correlation(c,d)))=\frac{1}{9}·\frac{1}{50-3}=\frac{1}{423}$$

From here you could end your test by hand, since the difference of both means will be normally distributed with mean 0 and variance double of each one:

$$var(mean(correlation(a,b))-mean(correlation(c,d)))=2·\frac{1}{423}$$

And you can use that to compute the p-value using difference of z values.

Anyway, we could also use the calculator, but we'll need to tell it a little lie.

We need to notice that the calculator just uses N to compute the variance of z values, because z values are just function of r, not N. Therefore, if the calculator is given the means of the correlations it will compute z values correctly.

After getting the two z values the calculator just finds the p-value assuming that both z values are normally distributed with the same mean and variance $\frac{1}{N-3}$. That's just what we want it to do, except that we know that variance is not $\frac{1}{N-3}$ but $\frac{1}{423}$ (because we are not giving the calculator two correlation but two means of 9 correlations). Here comes our little lie: we tell the calculator that $N=426$ and it will use variance $\frac{1}{426-3}=\frac{1}{423}$ and return the correct p-value.

In this answer I made several assumptions that I'd like to discuss:

• Independence of $correlation(a,b)$ and $correlation(c,d)$: if they are correlated, both z values will be correlated too, and it will need to be taken in account in the distribution of difference of p-values. The challenge might be estimate correlation between correlations (because of small sample of correlations: 9) and to transform that correlation to correlation between z values (that might not be straightforward). Anyway, unaccounted lack of independence would produce a lower variance and and higher p-value. Therefore, if you reject null hypothesis assuming independence, you would reject it even if lack of independence is taken in account; the opposite is not true.
• Null hypothesis is that all correlations are the same: For me, that is the most obvious interpretation of the question. However, another possible interpretation is that the mean of $correlation(a,b)$ equals the mean of $correlation(c,d)$. That would just be a two means test that could be solved using a t-test if we can made some assumptions on the distribution on correlations or otherwise can be attempted using a non parametric test.
• I believe your answer is incorrect. Seriously, read reference I provided. You do not obtain proper CI using the Fisher transformation. I used to think that what you are using is correct, just like you do. However, it failed simulation miserably.
– Carl
Commented Oct 19, 2016 at 21:46
• @Carl I can't read the reference, I can just read the abstract. However: 1) The abstract discourages using significance tests here (while acknowledging that's the usual practice), but the OP clearly asked about how to make a significance test using r-z transform. 2) That some source proposed a different tool than the usual one doesn't make this one wrong. 3) The main difference between your interesting answer and mine are assumptions, which we can't actually know if hold for this particular problem. 4) If my answer is wrong, please tell where.
– Pere
Commented Oct 19, 2016 at 22:00
• @Carl In short: The OP asked about a particular problem. I tried to help to solve this problem while you tried to address the OP needs by avoiding this problem. Maybe your answer eases the OP's life a lot while mine is just a funny algebra game, but I don't see where any of us is wrong.
– Pere
Commented Oct 19, 2016 at 22:02
• Significance test using r-z transform does not work. The variance is too inaccurate to be used that way.
– Carl
Commented Oct 19, 2016 at 22:05
• Do you mean it doesn't work for this particular problem due to sample size or something? Or do you just mean that all people using and recommending it (like the calculator the OP linked) is wrong while you are right? Anyway, the question is about that significance test, and answering about it is correct. If somebody ask about how to ride a bicycle, telling him that he can go faster by car might be useful, but it doesn't answer the question.
– Pere
Commented Oct 19, 2016 at 22:25

The answers you will get will be incorrect as the calculator is assuming uncorrelated correlation coefficients. Next problem, the Fisher transform probably will not give you what you think it will.

Not to panic. I believe the answers you need are contained in Zou GY. Toward using confidence intervals to compare correlations. Psychological methods. 2007;12:399. You will have to read this with understanding, track down the appropriate reference, and tailor the result to your specific problem. But, at least, I think you will then have the correct approach.

Here are some of the equations: