1
$\begingroup$

In case I have two models, one of them has correlation factor of 0.7 and the other one has 0.85 how can I compare them? Is it meaningful to say that there is a relative increase of (0.85-0.7)/0.7 = 0.21 in the correlation factor? Or makes more sense to compare the absolute difference: there is an increase of 0.85-0.7 = 0.15 in the correlation?

Also, is an absolute increase of 0.15 good?

$\endgroup$
2
  • $\begingroup$ Do you mean correlation coefficient ? $\endgroup$ Commented Jun 18, 2016 at 12:05
  • $\begingroup$ Whether the difference is "good" will depend on several things including context and sample size. You could calculate a confidence interval for the difference, perhaps by bootstrapping or by the general formula for a confidence interval of a normally distributed variable (using Fisher's Z transform and taking the difference between the Z scores) $\endgroup$ Commented Jun 18, 2016 at 12:33

1 Answer 1

3
$\begingroup$

0.7 and 0.85 are both strong correlations. One question is whether they are different at all from a statistical perspective, or whether the observed differences could be just due to random sampling.

Assuming that you have Pearson's correlation coefficients from a sample, then you can construct a confidence interval for their difference based on the general formula for a confidence interval of a normally distributed variable:

$$ CI = X \pm Z_{(1-\alpha/2)} SE $$

where $X$ is the point estimate of the statistic of interest (which in your case will be the difference in the sample correlations), $\alpha$ is the confidence level, $Z_{(1-\alpha/2)}$ is the $(1-\alpha/2)$ percentile of the normal distribution and $SE$ is the standard error of the statistic of interest.

If this interval contains zero then you can conclude that there is no statistically significant difference, at the $\alpha\%$ level, between the two correlations. In other words, putting it simply, it is plausible that the actual difference between the two correlations is zero.

If the confidence interval does not contain zero then you can conclude that they are statistically significant, at the $\alpha\%$ level. Whether this difference is meaningful in a clinical/practical sense is a subjective question that depends on context and can't really be answered statistically.

Since correlations are bound in the interval $[-1,1]$ and their sampling distribution is not normal, they must be transformed first. This can be achieved using the inverse hyperbolic tangent (atanh) otherwise known as Fisher's Z transform:

$$ Z_1 = \text{atanh}(r) = \frac{1}{2} \ln(\frac{1+r_1}{1-r_1}) $$ $$ Z_2 = \text{atanh}(r) = \frac{1}{2} \ln(\frac{1+r_2}{1-r_2}) $$

where $r_1$ and $r_2$ are your sample correlation coefficients.

The standard error for the difference is

$$ SE_{(Z_1 - Z_2)} = \sqrt {\frac{1}{N_1 - 3}+ \frac{1}{N_2 - 3}} $$

where $Z_1$ and $Z_2$ are the transformed variables and $N_1$ and $N_2$ are the respective sample sizes. Then apply the confidence interval formula:

$$ CI = (Z_1-Z_2) \pm Z_{(1-\alpha/2)} SE_{(Z_1 - Z_2)} $$

If you wanted a 95% confidence interval, this would be:

$$ CI_1 = (Z_1-Z_2) + 1.96 SE_{(Z_1 - Z_2)} $$ $$ CI_2 = (Z_1-Z_2) - 1.96 SE_{(Z_1 - Z_2)} $$

Once the interval has been calculated on the $Z$ scale it can be transformed back to the correlation scale using the $\tanh$ function:

$$ \eta_1 = \tanh(CI_1) = \dfrac{\exp(2 CI_1)-1}{\exp(2 CI_1)+1}$$ $$ \eta_2 = \tanh(CI_2) = \dfrac{\exp(2 CI_2)-1}{\exp(2 CI_2)+1}$$

And then the confidence interval for the difference in correlations is $[\eta_1,\eta_2]$.

$\endgroup$
5
  • $\begingroup$ Thanks, that was really helpful. In this case, I should collect various correlation factors and use it to estimate the confidence interval. But what if I had just a single correlation coefficient pair (one for each model)? $\endgroup$
    – rph
    Commented Jun 19, 2016 at 7:25
  • $\begingroup$ @rkioji my answer is based on having one pair of correlations. You take the Fisher transform of each and use the difference between them in the general confidence interval formula, along with the standard error, before back-transforming to the correlation scale. $\endgroup$ Commented Jun 19, 2016 at 7:31
  • $\begingroup$ Sorry to ask this, but doesn't confidence interval make only sense when there are plenty of samples? $\endgroup$
    – rph
    Commented Jun 19, 2016 at 7:33
  • $\begingroup$ @rkioji no, you only need 1 sample (or in your case 2 samples since we are dealing with a difference). It's basically the same as calculating a confidence interval for the mean from a single sample. $\endgroup$ Commented Jun 19, 2016 at 7:38
  • $\begingroup$ Very good, +1. Diedenhofen & Musch (2015, PLoS ONE) give more information, although their cocor R package appears to have disappeared. $\endgroup$ Commented Jun 19, 2016 at 9:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.