Confidence interval of correlation coefficient with different null hypothesis

Question

Suppose that I have two datasets

import pandas as pd
import numpy as np
import scipy.stats as st

x = [10, 15, 20, 50, 100]

y = [20, 31, 38, 51, 97]

I know that it is possible to compute the correlation $\rho$ between these sets and a 95% confidence interval with a Fisher transform, as follows:

r, p = st.pearsonr(x,y)

r_z = np.arctanh(r)
se = 1/np.sqrt(x.size-3)

alpha = 0.05
z = st.norm.ppf(1-alpha/2)

lo_z, hi_z = r_z-z*se, r_z+z*se

lo, hi = np.tanh((lo_z, hi_z))

However I have two questions:

• I believe, in the above methodology, the underlying null hypothesis is $\rho = 0$ and the alternative hypothesis is $\rho \neq 0$. However, in my case, the prior is that the two series $x,y$ are correlated with $\rho = 1$ and my alternative is that $\rho \neq 1$

• Second point, the size of my data are usually quite small, say between 4 and 15 data points. Given that I know the Fisher transform is an asymptotic approximation for large $n$, is there a better method than above for calculating the confidence interval I want?

Answer 1

If the true population proportion is exactly $\rho = 0,$ then the sample of $(x.y)$ pairs in your Question could not occur because your points don't all lie exactly on a line. Maybe there is a different way to pose your problem in which it makes sense to get a lower bound on $\rho$ based on the kind of data you have.

I think an additional difficulty is that the usual Fisher test and CI assume normal data, which it seems you don't have. The first thing that comes to mind is to get a nonparametric bootstrap CI, but useful bootstrapping needs larger samples than you may have.

My crude quantile-method bootstrap for the example in your question is as follows:

x = c(10, 15, 20, 50, 100)
y = c(20, 31, 38, 51, 97)
B = 10^5;  r.re = numeric(B)
ix = 1:5
set.seed(2020)
for(i in 1:B) {
  ix.re = sample(ix, 5, rep=T)
  r.re[i] = cor(x[ix.re], y[ix.re]) }
There were 50 or more warnings (use warnings() to see the first 50)
quantile(r.re, .05, na.rm=T)
      5% 
0.927494 

So the crude bootstrap 95% lower bound is about 0.93. I don't know the details of your project, so I'll leave it to you to decide whether such a bootstrap bound would be useful.

Notes: (1) The warnings occur because occasionally the same $(x,y)$ pair is chosen five times, leaving the sample correlation undefined on account of a 0 denominator. (Yet another difficulty with such small samples.) There were 164 NAs out of $B =100\,000$ re-samples---very nearly the expected number $B/5^4 = 160.$

summary(r.re)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
 0.8805  0.9840  0.9911  0.9830  0.9956  1.0000     164 

(2) A two-sided 95% bootstrap CI is $(.91, 1),$ but the upper value occurs because there are enough instances in which the re-sample has only two unique $(x,y)$ pairs.

quantile(r.re, c(.025, .975), na.rm=T)
     2.5%     97.5% 
0.9082498 1.0000000 

mean(r.re == 1, na.rm=T)
[1] 0.08647181

(3) A run with $B = 10^6$ yielded essentially the same results. With both run lengths there were only 84 uniquely different values of r.re.

length(unique(r.re))
[1] 84