6
$\begingroup$

The p-values provided by the built-in Matlab and Pylab Pearson correlation functions are stated to be inaccurate for small sample sizes, or when the samples are not normally distributed. The python documentation suggests using an N>500 for the p-value to be accurate, while the Matlab documentation gives no specific cutoff other than "large".

Does anyone know a method that can correctly test for statistical significance of the Pearson correlation coefficient under such circumstances?

My instinct is that I could just perform a permutation test by scrambling the X or Y values and re-sampling the Pearson correlation coefficient from the scrambled data -- then use that distribution to get a confidence interval. But, would this be correct?

Improve this question
$\endgroup$
3
  • $\begingroup$ Must you use the Pearson correlation coefficient or are you open to other types of correlation coefficients? $\endgroup$
    – TrynnaDoStat
    Jan 20, 2015 at 22:57
  • $\begingroup$ Other correlation coefficients should be fine. $\endgroup$
    – Michelle
    Jan 20, 2015 at 23:46
  • 1
    $\begingroup$ Oh, here's a question I should have asked before: how did you decide your data were non-normal? Or is the question only on the basis that there's the potential for non-normality (as suggested by the title)? (It's a wise question even so; there's always that potential, and sometimes it matters a lot.) $\endgroup$
    – Glen_b
    Jan 20, 2015 at 23:56

2 Answers 2

Reset to default
2
$\begingroup$

In terms of the p-value, the answer can be found in an earlier post. Basically, use the permutation test for n<20. A generally normalizing transformation, such as rankit, will work for larger n's and will be more powerful (Bishara & Hittner, 2012). Of course, if you transform, you're no longer looking at the linear relationship on the original scale.

In terms of the confidence interval, the answer is less clear. There aren't many published large-scale Monte Carlo comparisons. Puth et al. (2014) have some evidence that the Fisher Z can be inadequate with large violations of normality. There was no general solution - even bootstrapping with BCa did not solve it. You might consider either:

a)Spearman CIs with Fisher Z. Instead of using $SE_z=1/\sqrt(n-3)$, use the Fieller et al. (1957) estimate of standard error for the Fisher Z: $SE_z=1.03/\sqrt(n-3)$

b)Transforming via rankit, and then using the Fisher Z for the CI as usual

References:

Bishara, A. J., & Hittner, J. B. (2012). Testing the significance of a correlation with non-normal data: Comparison of Pearson, Spearman, transformation, and resampling approaches. Psychological Methods, 17, 399-417. doi:10.1037/a0028087

Fieller, E. C., Hartley, H. O., & Pearson, E. S. (1957). Tests for rank correlation coefficients. I. Biometrika, 44, 470-481.

Puth, M., Neuhäuser, M., & Ruxton, G. D. (2014). Effective use of Pearson’s product-moment correlation coefficient. Animal Behaviour, 93, 183-189.

Improve this answer
$\endgroup$
1
  • $\begingroup$ This is good news, the smallest sample I have is 19, and in fact it turns out that the approximations agree within 10% with the permutation tests. $\endgroup$
    – Michelle
    Jan 20, 2015 at 23:51
0
$\begingroup$

You could certainly perform a permutation test (of the null that the two are uncorrelated) in the manner you suggest, but you wouldn't normally "use that distribution to get a confidence interval" for the correlation.

You would instead use that distribution to get a p-value, or an acceptance (/rejection) region.

You could use another resampling approach - the bootstrap - to get an interval for the correlation, but that, too, is justified by large-sample arguments and may not get all that close to the desired coverage in small samples.

However, if your data are strongly non-normal, it's pretty common for the relationship between variables to be curved rather than linear. You might want to consider monotonic rather than linear association. In that case there are several measures of monotonic association (i.e. 'do the variables move up/down together?' type questions) that don't have normality assumptions required to test them.

What are you testing correlation for? (i.e. what's the aim here? What are you trying to figure out?)

Improve this answer
$\endgroup$
4
  • $\begingroup$ I have some small samples (N=20-50) that appear to be weakly linearly correlated by visual inspection. I'd like a rigorous way to summarize these results, and state for which conditions there is significant correlation. $\endgroup$
    – Michelle
    Jan 20, 2015 at 23:47
  • 2
    $\begingroup$ You can certainly use a permutation test for the 'test for significance' part, though both the Kendall and Spearman tests have very good power against the linear-correlation alternative in a wide variety of circumstances (they're effectively permutation tests of association based on the ranks when the data are continuous). What sort of non-normality is this? Do you have discreteness, heavy skewness, or something else? Do you have count data? Likert scales? continuous proportions? $\endgroup$
    – Glen_b
    Jan 20, 2015 at 23:51
  • $\begingroup$ By the way, when I was asking "what's the aim here? What are you trying to figure out?" I was not seeking a an answer containing statistical jargon. Why test correlation? What's the underlying point? What would it tell you, subject-matter-wise? That may lead to much better advice. $\endgroup$
    – Glen_b
    Jan 20, 2015 at 23:54
  • $\begingroup$ While I stand by the statement " if your data are strongly non-normal, it's pretty common for the relationship between variables to be curved rather than linear" that should not be taken to imply that if the variables are marginally normal that you should expect the correlation to be linear. $\endgroup$
    – Glen_b
    Apr 18 at 5:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.