2
$\begingroup$

What is the explanation for obtaining a Pearson's correlation coefficient value that is significantly larger (a factor of ~2) than the Spearman's rank correlation coefficient value (on the same data)?

Doesn't this goes against the idea that Spearman's rank correlation coefficient, being the Pearson's correlation coefficient of the ranked data, can be seen as a generalization of Pearson's evaluation for monotonic dependences instead of linear ones? How can the correlation coefficient value for the monotonic dependence be smaller than that for a linear dependence only?

I was surprised to see that this was possible in a dataset with $N$~100 elements. I should add that the p-value associated to the Pearson's correlation coefficient is of 0.0 while that of Spearman's rank is of ~0.10.

Possible explanation:

This behaviour might be driven by the extreme values of the dataset. I compare the values of Pearson's c.c. ($\rho$) and Spearman's rank c.c. ($\rho_r$) after removal of these. I present the 2-sided p-values.

  • Full dataset: $\rho$ = 0.381 (p-value: 0.000), $\rho_r$ = 0.151 (p-value: 0.131)

  • One outlier removed: $\rho$ = 0.336 (p-value: 0.001), $\rho_r$ = 0.125 (p-value: 0.213)

  • Three outliers removed: $\rho$ = 0.167 (p-value: 0.100), $\rho_r$ = 0.076 (p-value: 0.459)

The remaining distribution (plotted) does not seem affected by the presence of outliers and yet is still exhibits the same behaviour. The full data is available here; note the outliers correspond to the first three rows.

Distribution after the three outliers are removed (the first three rows in the attached file)

$\endgroup$
  • $\begingroup$ Take any bivariate dataset. Plot it. Pick any one of its points and slide it up and to the right until it is so far from the rest of the points that you cannot distinguish them from each other on the graph--they are all clustered in a little space at the lower left. (1) What happens to the Spearman coefficient? (2) What happens to the Pearson coefficient? $\endgroup$ – whuber Jun 15 '16 at 15:17
  • $\begingroup$ Thank you for the prompt answer. I looked at this by removing first one outlier in the extreme right, and then the other two. I update my question with it. The results remains largely unchanged. $\endgroup$ – pedrofigueira Jun 15 '16 at 15:44
  • $\begingroup$ On the contrary--if the Pearson coefficient remains largely unchanged, you simply haven't moved the point sufficiently far. (Note that I wrote moved, not removed!) Name any value between $-1$ and $1$ you wish: there will be a place you can put a single point that will make the Pearson coefficient equal to your chosen value. $\endgroup$ – whuber Jun 15 '16 at 16:21
  • 1
    $\begingroup$ That property is the entire point of Spearman's coefficient. By using ranks, no individual point out of $n$ can move beyond the square bounded by $1$ and $n$ in both coordinates. This limits the influence any point can have on the coefficient. That should suffice for an "intuitive explanation," but if you are seeking an explanation based in the meaning and characteristics of your particular data, then you will need to disclose that information in your post. $\endgroup$ – whuber Jun 15 '16 at 17:34
  • 1
    $\begingroup$ The Spearman correlation is just the Pearson correlation of the ranks. All ranks must lie between $1$ and $n$. $\endgroup$ – whuber Jun 16 '16 at 13:48
6
$\begingroup$

This is a simple dataset, where the points come alternating from two linear functions: the raw data

The pearson correlation detects, there is a general upwards motion in the combined data (red an black together) and is r=.453 The spearman correlation just sees the ranks, which are distributed like this: the ranks of the above data

There is a high and a low rank alternating, so no clear trend for spearman. Spearman r = .079 This pearson is 5.7 times as high and you can easily increase that value by extending the row. You can even easily get a negative Spearman for a positive Pearson by just leaving out the last value. So there is nothing in the way of a compbination of a large Pearson and a small Spearman r and the above picture is even a bit similar to your's.

You can easily see how I constructed the data by looking at them:

1, -.01, 2, -.02, 3, -.03, 4, -.04, 5, -.05, 6, -.06, 7, -.07, 8, -.08, 9, -.09, 10

Hope that helps, Bernhard

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.