# Interpreting the results of different correlation methods

I assessed the correlation between two groups of values, A and B. I used both the Spearman correlation and the Chatterjee correlation. There are four potential cases to observe in this correlation assessment.

Case Chatterjee Spearman
1 Not-Significant Not-Significant
2 Not-Significant Significant
3 Significant Not-Significant
4 Significant Significant

My understanding is as below:

• Case 1: We can conclude that the concepts do not have a statistically significant relationship.
• Case 2: I do not know.
• Case 3: Concepts have a non-linear relationship depending on the correlation coefficient magnitude.
• Case 4: Concepts have a linear relationship, as Spearman is significant, again depending on the correlation coefficient magnitude.

How can we interpret the Case-2? Are my interpretations for the other cases correct?

• Do you consider correlation as a binary concept (significant or not)? I would think it is a continuous measure. Statistical significance only says that the confidence interval for the coefficient does not include zero, which will always happen if you collect more data. Jun 8, 2023 at 5:33
• Thank you for your interest @cdalitz. I could not understand your answer. Could you give more details on your comment? Jun 8, 2023 at 6:46
• I have elaborated on my comment in an answer and also given an example where xicor and cor yield considrably different values. If this answers your question, please accept the answer so that the thred is no longer marked as unanswered. Jun 25, 2023 at 10:29

Correlation is not a binary yes/no property, but a continuous feature that can be weak or strong, which is indicated by the value of the correlation coefficient. Testing for "statistical significance" does not answer the question whether there is a correlation or not, but instead whether you have collected enough data so that the confidence interval for your correlation estimate does not include zero, or -in other words- whether you can be somewhat confident that the correlation is not exactly zero. This may happen even when the correlation is extremely weak. I would thus recommend not to report merely "statistical significance" or a p value, but a confidence interval for the correlation coefficient.

You will find a lot of background with an internet search for "null hypothesis testing controversy". One particular problem with your approach of merely testing for statistical significance is that a correlation of exactly zero rarely (some would argue: never) occurs in practice. This means that you can always make a correlation "significant", however small it may be, simply by collecting more data. Actually however, you do not make a correaltion "significant" (in its colloquial meaning) by collecting more data, but you make the estimated correlation value more accurate so that zero drops out of the confidence interval.

After this caveat, let me address your original question. The hypothesis tests for non-zero Pearson correlation and for non-zero Chatterjee correlation are different and have different power, which depends on the underlying data generating process. "Power" means the probability of a test to reject a null hypothesis when it is false. In Fig. 5 on p. 13 of Chatterjee's paper you will find Monte Carlo simulations of the power of different independence tests, and Chatterjee's test has the lowest power in many situations, but the highest power in one situation. This means that it may well be that one test reports a significant result and another a non-significant result for the same degree of correlation.

Concerning Chatterjee's correlation, beware that it is only an asymptotic measure and, unlike Pearson's correlation, it has a considerable pessimistic bias for small n, even for exactly linearly correlated data:

> library(XICOR)
> x <- 1:5
> xicor(x,x)
[1] 0.5
> cor(x,x)
[1] 1


In this case, the number of data points is so small that this is also an example for your Case 2, if you use the usual 5% significance level:

> cor.test(x,x)$$p.value [1] 3.971862e-24 > xicor(x,x,pvalue=T)$$pval
[1] 0.05237874


Edit: Concerning Chatterjee's correlation, it turns out that constructing confidence intervals is a bit intricate because the usual n-out-of-n booststrap fails in this case and the resulting confidence intervals have zero coverage probability. It is possible, however, to construct confidence intervals with the m-out-of-n bootstrap, as is described in this paper (it also describes a bias correction factor that fixes the first example in this answer above): https://arxiv.org/abs/2312.15496