Is there a reliable way to say if two Pearson correlations from the same sample (do not) differ significantly? More concrete, I calculated the correlation between a total score on a questionnaire and an other variable, and a subscore of the same questionnaire and the variable. The correlations are respectively .239 and .234, so they look quite similar to me. (The other two subscales did not significantly correlate with the variable). Could I use a fisher Z to check if the two correlations indeed do not significantly differ, or is the fact that they are not independent a problem?
$\begingroup$
$\endgroup$
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
2
Firstly I would point out that these correlations are fairly low.
Second, have you plotted the data to investigate possible non-linear associations?
Third, I would say that common sense should dictate that correlations of 0.239 and 0.234 are essentially the same and searching for a test to confirm this, unless the sample size is absolutely enormous, is folly.
Fourth, you could calculate confidence intervals for both statistics, and if they do not overlap, then you can conclude that they are statistically significantly different. However, this would be invalid since the 2 samples are not independent. Moreover, as per my third point, even if you did have such an enormous sample and a test which validly concluded that a significant difference exists, I would find it hard to belive that the difference was practically significant.
answered Apr 13, 2019 at 10:32
-
$\begingroup$ Thanks for your reply. I am indeed aware that the correlations are small. I have checked for non-linear associations. I also feel like the difference is not meaningfull, but I just wanted make I do everything in the best possible way. Thanks! $\endgroup$– ChaFoCommented Apr 14, 2019 at 13:07
-
$\begingroup$ @ChaFo that's OK, but you don't always have to make a formal test, especially where it seems obvious that they are essentially the same. How many observations do you have? $\endgroup$ Commented Apr 14, 2019 at 13:12
$\begingroup$
$\endgroup$
3
Expanding on Robert Long's answer (+1 to Robert) I'd say that testing for a difference between these is folly, regardless of sample size. Look! Is 0.239 different from 0.234? Well, maybe it is. There are situations where a very small effect size is very important. If a plane crashes 1 in 1,000 flights, that's a big big problem. I can't think, offhand, of a situation where this tiny difference in correlations could be meaningful, but maybe there is one. Whether it is significant or not is not the point.
Also, the dependence will surely be a problem. If you really wanted to see something like this, I'd find a third correlation: The correlation between the test after removing the subtest. Then you can compare that to the correlation with the subtest.
Finally, it's unclear to me what you are trying to show, but I think you are trying to show that these are not different. In that case, the usual null hypothesis tests are inappropriate. You should be looking at tests of equivalence (if, in fact you want to look at significance at all).
answered Apr 13, 2019 at 12:06
-
1$\begingroup$ Excellent points, Peter (+1) $\endgroup$ Commented Apr 13, 2019 at 12:37
-
1$\begingroup$ Peter Flom, the population perspective in epidemiology says, in effect, that a tiny change in risk—one that is so small as to be effectively inconsequential clinically—is a big deal if it is multiplied across an entire population. Changing someone's risk of stroke by 1 in 10,000 per year is kinda meh. Changing 10,000,000 people's risk of stroke by 1 in 10,000 is a change of a 1,000 strokes per year: a big deal. See Rose, G. (1985). Sick individuals and sick populations. International Journal of Epidemiology, 14(1), 32–28. $\endgroup$– AlexisCommented Apr 13, 2019 at 18:00
-
$\begingroup$ Of course, Pearson's correlation coefficient alone isn't likely to be the most used measure of contrasts in risk, but I think small associations can matter. $\endgroup$– AlexisCommented Apr 13, 2019 at 18:01
$\begingroup$
$\endgroup$
3
Yes, it is possible to perform a significance test using the Fisher transform. This also depends on $N$, the number of samples used to compute the Pearson correlations. This blog post describes the method in more detail, and provides R code for it.
-
3$\begingroup$ Your reference is inappropriate for comparing correlation coefficients that share data, as is the case here. The OP points out that "the fact they are not independent" is the problem. $\endgroup$– whuber ♦Commented Apr 13, 2019 at 15:47
-
1$\begingroup$ Yes, I see. OP's situation involves overlap between the two datasets, but is not a case of paired data. Therefore, my answer is inappropriate. $\endgroup$– Bai LiCommented Apr 13, 2019 at 16:05
-
1$\begingroup$ Actually, it sounds like the data are triples: that's what makes it possible to compute more than one correlation coefficient. $\endgroup$– whuber ♦Commented Apr 13, 2019 at 16:12