What is the correct way to define confidence intervals for the mean of multiple correlations? I understand how to calculate CIs for individual correlation coefficients, I also understand how to calculate the mean correlation through Fisher's transformation. But what are the confidence intervals for this mean correlation value? Surely, it can not be just the +-(respective z-score of CI level) multiplied by SE of the final mean Fisher's transformed correlation score, since each single Fisher's transformed correlation score is constructed from multiple data and has it's own precision and SE.

  • $\begingroup$ This question raises two issues: first, what would the mean of a bunch of correlation coefficients estimate? Second, how are these correlation coefficients related? Are they independent or not; and if they are not independent, we need the details of how they are generated. Typically the correlation coefficients estimated from a single multivariate dataset are strongly interdependent and it would be erroneous to work with them as if they were independent. $\endgroup$
    – whuber
    Nov 23 '20 at 15:05
  • $\begingroup$ I wanted to keep the question simple, but I understand some clarification would be needed for a precise answer. My goal is to provide a simple yet convincing argument that in a correlation matrix of supposedly consistently correlated variables, certain cells contain higher correlation coefficients than different cells, on average. $\endgroup$ Nov 23 '20 at 16:05
  • $\begingroup$ For any such argument to be valid, it would need to appeal to how the data were collected and how their joint distribution is modeled. $\endgroup$
    – whuber
    Nov 23 '20 at 16:15
  • $\begingroup$ If it helps, we are talking about Likert scale answers, range 1-6, on a one-factor psychological questionnaire with high internal consistency (all questions have similar content, factor loading, and are all moderately to strongly positively correlated, range circa 0.5 - 0.8). $\endgroup$ Nov 23 '20 at 16:23
  • $\begingroup$ That raises additional complications, because there is now the additional issue of what a correlation coefficient even means, given that these are just ordinal scales and not numerical ones. It also suggests you are trying to solve some kind of unstated statistical problem with this averaging; and, because this averaging procedure is so unusual, you should forgive us for thinking there likely is a better solution to the underlying problem. $\endgroup$
    – whuber
    Nov 23 '20 at 16:26

Actually, what you describe is just fine.

If $\bf{x}$ $= [x_1, x_2, \dots, x_n]$, your set of data points, is approximately normally distributed, $\text{mean}(x) \pm \alpha \times\text{se}(x)$ produces a valid confidence interval, with $\alpha \approx 1.96$ for a 95% CI. As you note, since correlation coefficients aren't typically normally distributed, you can transform them using the Fisher z-transformation, calculated the mean and CI of the transformed values, and then back-transform to the original scale.

You're right that this is throwing away information: each of the coefficients has a different mean and SE, and you could obtain a better estimate by giving more weight to the more precisely-estimated correlations. This can be done, for instance, by fitting a multilevel model. However, the simple approach still yields calibrated confidence intervals.

The same thing happens when you want to calculate confidence intervals for a grand mean (a mean of means), for instance when you have multiple data points per group, and want to calculate a mean across groups.

Update: It's just occurred to me that you could, comments about the assumptions of your analysis notwithstanding, reframe this problem as a linear mixed model.

Let's say you currently have $n$ independent variables, $\bf x_1, x_2, \dots, x_n$, and $n$ dependant variables $\bf y_1, y_2, \dots, y_n$, and you're currently calculating $n$ correlation coefficients, $r_1, r_2, \dots, r_n$. As noted in the comments, I assume you believe that these correlations are independently sampled from a common distribution, and you're interested in calculating the mean and standard error of that distribution, for instance that you've collected $x$ and $y$ for each of the $n$ participants in an experiment, calculated $r$ for each participant, and wish to estimate $r$ at the population level.

This can be recast as a linear mixed model

$$y_i = \alpha_p + \beta_p x_i$$

where $\alpha_p$ and $\beta_p$ are the intercept and slope term for participant $p$, and $\alpha_p \sim \text{Normal}(A, \sigma_A)$, $\beta_p \sim \text{Normal}(B, \sigma_B)$. You're interested in estimating $B$, the population-level fixed effect for the slope parameter.

This can be done easily using lme4 by pivoting your data to a long format with one row per observation and columns for x, y, and participant, and using the command

lmer(y ~ 1 + x + (1 + x|participant), data=your_long_data)

Good luck!

  • $\begingroup$ "each of the coefficients has a different mean and SE, and you could obtain a better estimate by giving more weight to the more precisely-estimated correlations" This is more specifically only the case when the assumption is that all the correlations are sampled from the same distribution. To me it is not clear whether this is the situation in this question. $\endgroup$ Nov 23 '20 at 13:25
  • $\begingroup$ Surely if they're calculating the mean correlation, and the standard error of the mean, they're already assuming that the correlations are sampled from the same distribution? $\endgroup$
    – Eoin
    Nov 23 '20 at 14:27
  • 1
    $\begingroup$ For me that is not sure. It could also be the mean of correlation coefficients for entirely different populations, or the mean of correlation coefficients which are different due to a random effect. If the different correlation coefficients are really considered to be the same then it would definitely be important to apply different weights in computing the mean and using the plain mean as you suggest in the first paragraph would be a bad advise for general situations. $\endgroup$ Nov 23 '20 at 14:47
  • $\begingroup$ This answer seems to assume all the correlation coefficient estimates are independent. $\endgroup$
    – whuber
    Nov 23 '20 at 15:06
  • $\begingroup$ It is the amount of precision I am losing I am interested in. I fear that when we construct CIs around mean correlation with this simple approach, the CIs would be quite wider than they should be. Because, here N = number of correlations, and it's relatively small compared to N = number of observation per variable, which is usually much higher and which we use to construct CIs around single correlation. But it is just my feeling and I might be wrong. $\endgroup$ Nov 23 '20 at 17:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.