I have a bunch of related datasets. The pearson correlations between pairs of them are typically definitely larger than the spearman correlations. That suggests any correlation is linear, but one might expect that even if the pearson and spearman were the same. What does it mean when there is a definite gap between the pearson and the spearman correlation and the pearson is larger? This seems to be a consistent feature across my datasets.
$\begingroup$
$\endgroup$
1
asked Apr 25, 2012 at 20:17
-
$\begingroup$ Very similar question with great answer here $\endgroup$– Colin T BowersOct 29, 2015 at 5:13
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
1
The Spearman correlation is just the Pearson correlation using the ranks (order statistics) instead of the actual numeric values. The answer to your question is that they're not measuring the same thing. Pearson: linear trend, Spearman: monotonic trend. That the Pearson correlation is higher just means the linear correlation is larger than the rank correlation. This is probably due to influential observations in the tails of the distribution that have large influence relative to their ranked values. Tests of association using the Pearson correlation are of higher power when the linearity holds in the data.
-
1$\begingroup$ I knew that spearman was just pearons on the ranks. I need to see whether there can be any other routes that could cause this as well, but influential tail observations that are more linearly correlated than the bulk of the data, but which lose their influence when replaced with their rankings would definitely cause what I am seeing. $\endgroup$ Apr 25, 2012 at 20:49
$\begingroup$
$\endgroup$
2
The Pearson Correlation assumes several assumptions for it to be accurate: 1) Each variable is normally distributed; 2) Homoscedasticity, the variance of each variable remains constant; and 3) Linearity, meaning that a scatter plot depicting the relationship shows data points clustering symmetrically around the regression line.
The Spearman Correlation is a nonparametric alternative to the Pearson one based on rank of the observations. The Spearman Correlation allows you to relax all three assumptions about your data set and derive correlations that are still reasonably accurate.
What your data implies is that it probably breaks materially one or more of the mentioned assumptions materially so that the two correlations differ significantly.
Given that you have a large gap between the two correlation you should investigate whether the variables of your data set are normally distributed, homoscedastic, and linear within a scatter plot.
The above investigation will facilitate your decision on whether the Spearman or the Pearson correlation coefficient is the more representative one.
-
2$\begingroup$ You are probably thinking of the associated inference test using the $t$-distribution, not the descriptive measure. However, this test is typically described as having different assumptions than the ones you stated. E.g., bivariate normality, a stronger assumption than individually normal distributions. Please provide references for your statements. $\endgroup$– caracalApr 25, 2012 at 22:12
-
4$\begingroup$ Incorrect. Inference on the Pearson correlation does not require any of these assumptions to hold. One could have a curvilinear relationship with heteroscedastic, non-normal data and the test of Pearson correlation (which is equivalent to inference on a linear regression model) is powered to detect the first order trend. The interpretation of Pearson correlation as the strength of first-order trend still holds. There are few circumstances where the first order trend measured by the Pearson correlation is not adequate for an analysis. $\endgroup$– AdamOApr 26, 2012 at 18:44