Suppose that I have two datasets
import pandas as pd import numpy as np import scipy.stats as st x = [10, 15, 20, 50, 100] y = [20, 31, 38, 51, 97]
I know that it is possible to compute the correlation $\rho$ between these sets and a 95% confidence interval with a Fisher transform, as follows:
r, p = st.pearsonr(x,y) r_z = np.arctanh(r) se = 1/np.sqrt(x.size-3) alpha = 0.05 z = st.norm.ppf(1-alpha/2) lo_z, hi_z = r_z-z*se, r_z+z*se lo, hi = np.tanh((lo_z, hi_z))
However I have two questions:
I believe, in the above methodology, the underlying null hypothesis is $\rho = 0$ and the alternative hypothesis is $\rho \neq 0$. However, in my case, the prior is that the two series $x,y$ are correlated with $\rho = 1$ and my alternative is that $\rho \neq 1$
Second point, the size of my data are usually quite small, say between 4 and 15 data points. Given that I know the Fisher transform is an asymptotic approximation for large $n$, is there a better method than above for calculating the confidence interval I want?