Exact central confidence interval for a correlation

Question

I am working my way through (T.J. Diccio & B. Efron, "Bootstrap Confidence Intervals", Statistical Science, 1996, 11(3), 189–228), and I'm stuck even before I get to the good stuff.

In the introduction, there is a table of before/after treatment values called cd4, and the correlation $\hat\theta$ is given as 0.723. The paper then states, "We can find an exact interval for $\theta$ if we are willing to assume bivariate normality ... The exact central 90% interval is (0.47, 0.86)."

Here I stipulate that I am a stats moron. When I key in the data and compute a confidence interval by hand in R, either using the Fisher transform or the T distribution, I don't get that interval. How does one compute the interval they are talking about?

> cd4
   subj base oneyr
1     1 2.12  2.47
2     2 4.35  4.61
3     3 3.39  5.26
4     4 2.51  3.02
5     5 4.04  6.36
6     6 5.10  5.93
7     7 3.77  3.93
8     8 3.35  4.09
9     9 4.10  4.88
10   10 3.35  3.81
11   11 4.15  4.74
12   12 3.56  3.29
13   13 3.39  5.55
14   14 1.88  2.82
15   15 2.56  4.23
16   16 2.96  3.23
17   17 2.49  2.56
18   18 3.03  4.31
19   19 2.66  4.37
20   20 3.00  2.40
> r = cor(cd4$base, cd4$oneyr)
> r
[1] 0.7231654

Fisher transform:

> se = 1/sqrt(17)
> se
[1] 0.2425356
> tanh(c(r-1.6448*se, r+1.6448*se))
[1] 0.3133382 0.8082940

T distribution:

> sr = sqrt((1-r^2)/(20-2))
> sr
[1] 0.1627936
> tc = abs(qt(0.05/2, 18))
> c(r-tc*sr, r+tc*sr)
[1] 0.3811486 1.0651821

Answer 1

You can get the same values with:

cor.test(cd4$baseline, cd4$oneyear, method = "pearson", conf.level = 0.9)

The method used to obtain such interval is explained in:

http://stat.ethz.ch/R-manual/R-patched/library/stats/html/cor.test.html

If method is "pearson", the test statistic is based on Pearson's product moment correlation coefficient cor(x, y) and follows a t distribution with length(x)-2 degrees of freedom if the samples follow independent normal distributions. If there are at least 4 complete pairs of observation, an asymptotic confidence interval is given based on Fisher's Z transform.

So, you can implement your own code to obtain C.I. by following these instructions if you wish to do so. See also:

http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient#Testing_using_Student.27s_t-distribution

Answer 2

You already mentioned that the Fisher transformation is not correct in your code. You first have to transform r to a z value (atanh part), then you add and subtract the standard error with the appropriate multiplier to get the correct confidence (as you did correctly). Finally, you have to transform the whole thing back into the r-metric (tanh part).

se <- 1/sqrt(17)
r <- 0.7231654
tanh(atanh(r)+c(1,-1)*qnorm(.95)*se)

Which results in

[1] 0.8650790 0.4740748

As mentioned in the comments, this is NOT the exact interval! To find an exact interval check out this work by Shieh: http://link.springer.com/10.1007/s11336-04-1221-6