6
$\begingroup$

I am comparing three different measures comprised of 10-20 items each. Each of these measures is actually a short form of a longer 40-item measure and each of the short forms shares some similar items, and thus they are quite similar. I am interested in seeing how each of these short forms are correlated with the same dependent variable.

I come up with r= .54, .56, and .58.

How can I assess whether these correlations are significantly different? It seems doing a simple r-to-z transformation would not be appropriate since the independent variables are quite similar and the dependent variable is in fact the same for all three correlations. As such, all of these *r*s overlap quite a bit.

I found this one paper:

Zou, G. (2007). Toward using confidence intervals to compare correlations. Psychological Methods, 12(4), 399.

But it seems way over my head and I am wondering whether there is an easier way to do it...

Any guidance would be appreciated!

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ My thinking is that those correlations are all too close to each other to detect a difference statistically for the given small sample size. $\endgroup$
    – Michael R. Chernick
    Commented May 29, 2012 at 20:18
  • 1
    $\begingroup$ Based on the abstract the Zou paper does look dead-on as one that would address your problems. If the fact that the 3 scores share some components makes you leery of using Fisher's Z transformation, the same problem would seem to apply to a regression approach. $\endgroup$
    – rolando2
    Commented Jan 26, 2013 at 1:10

2 Answers 2

Reset to default
1
$\begingroup$

You could fit a regression model with all three measures as predictors, then fit a new regression model with 1 or 2 dropped out and do a full-reduced model test to see if there is a significant difference in the models. This answers the question of "do the variables in the full, but not reduced, model contribute significantly above and beyond those in the reduced model?". As noted already given your sample size I doubt that you will see a difference, but this will give a p-value for those that feel the need for one.

A bit more meaningful may be to fit 3 regression model, each using one of the measures and your dependent variable, then plot the 3 pairwise scatterplots of the fitted(predicted) values. This is best done with an aspect ratio of 1 in a square plot and with an $y=x$ reference line. This can show that the 3 measures give essentially the same predictions, or if not can show where they differ.

Improve this answer
$\endgroup$
-1
$\begingroup$

I thought you were talking about less than 100. 4000 may be barely enough. but a difference of 0.02 is not very meaningful.

Improve this answer
$\endgroup$
2
  • $\begingroup$ CORRECTION: I actually have only about 84 cases. My previously conclusion resulted from a gross tabulation error (i.e., only 84 of my 4000 cases apparently have post scores, which I was not previously aware of). Regardless of the practical meaningfulness of the differences, I would like to know how I could calculate for statistical differences, for this paper and for the future. $\endgroup$
    – Behacad
    Commented May 30, 2012 at 0:07
  • 1
    $\begingroup$ I agree. We could answer your question for theoretical interest, but I think we can say with absolute confidence there is no way those small differences in estimated correlation will be statistically significant evidence of a difference in the real population, given your sample size. $\endgroup$
    – Peter Ellis
    Commented Sep 27, 2012 at 1:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.