How to get p-values or confidence intervals for pearson correlation coefficient when the sample is small and potentially non-Gaussian?

Question

The p-values provided by the built-in Matlab and Pylab Pearson correlation functions are stated to be inaccurate for small sample sizes, or when the samples are not normally distributed. The python documentation suggests using an N>500 for the p-value to be accurate, while the Matlab documentation gives no specific cutoff other than "large".

Does anyone know a method that can correctly test for statistical significance of the Pearson correlation coefficient under such circumstances?

My instinct is that I could just perform a permutation test by scrambling the X or Y values and re-sampling the Pearson correlation coefficient from the scrambled data -- then use that distribution to get a confidence interval. But, would this be correct?

Answer 1

In terms of the p-value, the answer can be found in an earlier post. Basically, use the permutation test for n<20. A generally normalizing transformation, such as rankit, will work for larger n's and will be more powerful (Bishara & Hittner, 2012). Of course, if you transform, you're no longer looking at the linear relationship on the original scale.

In terms of the confidence interval, the answer is less clear. There aren't many published large-scale Monte Carlo comparisons. Puth et al. (2014) have some evidence that the Fisher Z can be inadequate with large violations of normality. There was no general solution - even bootstrapping with BCa did not solve it. You might consider either:

a)Spearman CIs with Fisher Z. Instead of using $SE_z=1/\sqrt(n-3)$, use the Fieller et al. (1957) estimate of standard error for the Fisher Z: $SE_z=1.03/\sqrt(n-3)$

b)Transforming via rankit, and then using the Fisher Z for the CI as usual

References:

Bishara, A. J., & Hittner, J. B. (2012). Testing the significance of a correlation with non-normal data: Comparison of Pearson, Spearman, transformation, and resampling approaches. Psychological Methods, 17, 399-417. doi:10.1037/a0028087

Fieller, E. C., Hartley, H. O., & Pearson, E. S. (1957). Tests for rank correlation coefficients. I. Biometrika, 44, 470-481.

Puth, M., Neuhäuser, M., & Ruxton, G. D. (2014). Effective use of Pearson’s product-moment correlation coefficient. Animal Behaviour, 93, 183-189.

Answer 2

You could certainly perform a permutation test (of the null that the two are uncorrelated) in the manner you suggest, but you wouldn't normally "use that distribution to get a confidence interval" for the correlation.

You would instead use that distribution to get a p-value, or an acceptance (/rejection) region.

You could use another resampling approach - the bootstrap - to get an interval for the correlation, but that, too, is justified by large-sample arguments and may not get all that close to the desired coverage in small samples.

However, if your data are strongly non-normal, it's pretty common for the relationship between variables to be curved rather than linear. You might want to consider monotonic rather than linear association. In that case there are several measures of monotonic association (i.e. 'do the variables move up/down together?' type questions) that don't have normality assumptions required to test them.

What are you testing correlation for? (i.e. what's the aim here? What are you trying to figure out?)