8
$\begingroup$

I've been asked by a reviewer if the Pearson correlations (r-values) presented in a table can be compared to each other so one can claim that one is "stronger" than another (other than just eyeballing the actual r-values).

How would you go about this? I've found this method

http://vassarstats.net/rdiff.html

but am unsure if this applies.

$\endgroup$
  • 1
    $\begingroup$ Are your two correlations calculated from the same sample of observations? $\endgroup$ – Dimitriy V. Masterov May 22 '14 at 23:21
  • $\begingroup$ Something like a confidence interval for the difference could work. You might use simulation under some distributional assumption, or asymptotic normal approximations. In large samples, you could use bootstrapping. $\endgroup$ – Glen_b May 22 '14 at 23:26
  • $\begingroup$ AFAIK deriving the CI from the z scores and back-transforming z to r for a CI on rs works. Even for a bootstrap CI, it might be appropriate to first transform them. $\endgroup$ – jona May 23 '14 at 12:43
21
$\begingroup$

(I assume you're talking about r's obtained from a sample.)

The test on that website applies in the sense that it treats r like any parameter whose value may differ between two populations. How is r any different from any other measure, such as the mean, which you're very confident in comparing, using the t-test? Well, it's different in that it's bound between -1,1, it doesn't have the proper distribution, so you need to Fisher transform it before doing inference (and back transform it afterwards, if you want to e.g. get a CI). The z scores resulting from the test do have the proper form to do inference on. That's what the test you're linking to is doing.

So what you link to is a procedure of inferring what might happen if you could get the r for the entirety of the population(s) from which you're sampling - would the r for one group be higher than for the other, or would they be precisely the same? Let's call this later hypothesis H$_0$. If the test returns a low p value, it implies that based on your sample, you should have little confidence in the hypothesis that the true value for the difference between the two r's would be exactly 0 (as such data would occur rarely if the difference in r was exactly 0). If not, you do not have the data to reject, with confidence, this hypothesis of precisely equal r, either because it is true and/or because your sample is insufficient. Note that I could have done the same story about the difference in means (using the t-test), or any other measure.

A completely different question is if the difference between the two would be meaningful. Sadly, there is no straight-forward answer to this, and no statistical test can give you the answer. Maybe the true value (the population value, not the one you observe) of r is .5 in one, and .47 in the other group. In this case, the statistical hypothesis of their equivalence (our H$_0$) would be false. But is this a meaningful difference? It depends - is something on the order of 3% more explained variance meaningful, or meaningless? Cohen has given rough guidelines for interpreting r (and presumably, differences between r's), but did so only under the advice that these are nothing but a starting point. And you do not even know the exact difference, even if you do some inference, e.g. by calculating the CI for the differences between the two correlations. Most likely, a range of possible differences will be compatible with your data.

A comparatively safe bet would be computing the confidence intervals for your r's and possibly the CI for their difference, and letting the reader decide.

$\endgroup$
  • 3
    $\begingroup$ I don't know if you're a teacher, but you should be. This is easy to understand even for someone with basic knowledge of statistics thanks to your great explanation. $\endgroup$ – Andre Terra Mar 23 '17 at 22:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.