# Why is my replication of Silver & Dunlap 1987 not working out?

I'm trying to replicate Silver & Dunlap (1987). I'm just comparing averaging correlations or averaging z transform correlations and back transforming. I seem to not be replicating the asymmetry in the bias they find (back transformed zs are not closer to the population value for me than rs). Any thoughts? Is it possible that 1987 computing power just didn't explore the space enough?

# Fisher's r2z
fr2z <- atanh
# and back
fz2r <- tanh

# a function that generates a matrix of two correlated variables
rcor <- function(n, m1, m2, var1, var2, corr12){
require(MASS)
Sigma <- c(var1, sqrt(var1*var2)*corr12, sqrt(var1*var2)*corr12, var2)
Sigma <- matrix(Sigma, 2, 2)
return( mvrnorm(n, c(m1,m2), Sigma, empirical=FALSE) )
}


With these function it's easy to look at a bunch of correlations (basically replicate silver and dunlap 1987) and see the difference between averaging correlations and averaging z-scores and back transforming. Here's just one.

r <- 0.9
Y <- replicate(20000, rcor(10, 0, 0, 1, 1, r))
rs <- apply(Y, 3, function(x) cor(x[,1], x[,2]))
mean(rs) - r
zs <- fr2z(rs)
fz2r( mean(zs) ) - r


Just looking at the sample size of 10 and correlations of 0.1, 0.5, and 0.9 these are the results.

     rho  r bias   z bias
0.1  -0.006   0.006
0.5  -0.024   0.021
0.9  -0.011   0.011


And these are derived from Table 1 of Silver & Dunlap.

     rho  r bias   z bias
0.1  -0.007   0.003
0.5  -0.025   0.001
0.9  -0.011  -0.007


These are quite different results. From my test I'm seeing that it's just a matter of direction of bias, not magnitude. But, in the published paper they're finding much less magnitude with z. I couldn't find a published non-replication.

-
I'm stuck at your first two lines. They do not appear to be correct R syntax. They also appear to assume that atanh is its own inverse, but it's not: tanh is the inverse of atanh. –  whuber Feb 2 '12 at 1:05
They're just typos in the question... fixed. –  John Feb 2 '12 at 1:12
To me, just by eye, the r bias for rho of 0.5 in the Silver & Dunlap table looks like the outlier to me. I certainly can't vouch for the quality of the journal, which appears quite new and a bit rough around the edges, but I did find this recent paper with a Google search. See, in particular, their Table 3 which, again, by eye, appears to corroborate your results. –  cardinal Feb 2 '12 at 2:26
Thanks for the link cardinal. Perhaps make an answer so I can accept it. –  John Feb 2 '12 at 3:15
@whuber: You raise good points. I didn't have ready access to the S&D paper either, so my remarks have been reduced to conjecture. If we ever happen to meet in person, I'll swap a story or two with you over a beer on the frustrations of dealing with those who insist on averaging correlations. I agree wholeheartedly with your comments on the matter. That said, it may make sense in some settings that I'm generally less familiar with. :) –  cardinal Feb 2 '12 at 3:25

To me, the r bias entry for rho of 0.5 in the Silver & Dunlap table looks the most suspiciously different to me. However, that said, it does match your estimated value quite closely.

Unfortunately, I don't have access to the Silver & Dunlap paper at the moment, but a Google search did turn up a recent paper that performs a similar study to the one you've done. It is

R. L. Gorsuch and C. S. Lehmann (2010), Correlation coefficients: Mean bias and confidence interval distortions, Journal of Methods and Measurement in the Social Sciences, vol. 1, no. 2, 52–65.

See, in particular, their Table 3 which, at least by eye, appears to corroborate your results.

I certainly can't vouch for the quality of the journal (or the whole paper), which looks quite new and a bit rough around the edges, in my estimation. Caveat lector.

For an in-depth, more theoretical, treatment of inference on correlation (simple, partial, and multiple) primarily in a multivariate normal framework, a good reference is

F. A. Graybill, Theory and application of the linear model, Duxbury Press, 1976, Chapter 11.

It does not concern itself much with small-sample performance or applied aspects, though.

-