27
$\begingroup$

Maybe this question is naive, but:

If linear regression is closely related to Pearson's correlation coefficient, are there any regression techniques closely related to Kendall's and Spearman's correlation coefficients?

$\endgroup$
  • 3
    $\begingroup$ As a simple example where you have one explanatory and a dependent variable: A linear regression of the ranks of $x$ and $y$ would yield Spearman's correlation coefficient as regression coefficient. And in this case, $x$ and $y$ are interchangeable in the regression. $\endgroup$ – COOLSerdash Jul 20 '13 at 9:28
  • 2
    $\begingroup$ Just a few thoughts. Kendall's $\tau$ and Spearman's $\rho$ are both correlation coefficients based on ranks. The sought after relationship between $x$ and $y$ would then need to involve their ranks. However, computing the ranks introduces dependence between the observations, which in turn imposes dependence between the error terms, eliminating linear regression. However, in a different setting, modeling the dependence structure between $x$ and $y$ with copulas would make a link with Kendall's $\tau$ and/or Spearman's $\rho$ possible, depending on the choice of copula. $\endgroup$ – QuantIbex Jul 20 '13 at 10:35
  • 1
    $\begingroup$ @QuantIbex does that dependence necessarily imply $E[\varepsilon_i\varepsilon_j]\neq 0$? $\endgroup$ – shadowtalker Jul 31 '14 at 3:07
21
+50
$\begingroup$

There's a very straightforward means by which to use almost any correlation measure to fit linear regressions, and which reproduces least squares when you use the Pearson correlation.

Consider that if the slope of a relationship is $\beta$, the correlation between $y-\beta x$ and $x$ should be expected to be $0$.

Indeed, if it were anything other than $0$, there'd be some uncaptured linear relationship - which is what the correlation measure would be picking up.

We might therefore estimate the slope by finding the slope, $\tilde{\beta}$ that makes the sample correlation between $y-\tilde{\beta} x$ and $x$ be $0$. In many cases -- e.g. when using rank-based measures -- the correlation will be a step-function of the value of the slope estimate, so there may be an interval where it's zero. In that case we normally define the sample estimate to be the center of the interval. Often the step function jumps from above zero to below zero at some point, and in that case the estimate is at the jump point.

This definition works, for example, with all manner of rank based and robust correlations. It can also be used to obtain an interval for the slope (in the usual manner - by finding the slopes that mark the border between just significant correlations and just insignificant correlations).

This only defines the slope, of course; once the slope is estimated, the intercept can be based on a suitable location estimate computed on the residuals $y-\tilde{\beta}x$. With the rank-based correlations the median is a common choice, but there are many other suitable choices.

Here's the correlation plotted against the slope for the car data in R:

enter image description here

The Pearson correlation crosses 0 at the least squares slope, 3.932
The Kendall correlation crosses 0 at the Theil-Sen slope, 3.667
The Spearman correlation crosses 0 giving a "Spearman-line" slope of 3.714

Those are the three slope estimates for our example. Now we need intercepts. For simplicity I'll just use the mean residual for the first intercept and the median for the other two (it doesn't matter very much in this case):

           intercept
 Pearson:  -17.573 *     
 Kendall:  -15.667
 Spearman: -16.285

*(the small difference from least squares is due to rounding error in the slope estimate; no doubt there's similar rounding error in the other estimates)

The corresponding fitted lines (using the same color scheme as above) are:

enter image description here

Edit: By comparison, the quadrant-correlation slope is 3.333

Both the Kendall correlation and Spearman correlation slopes are substantially more robust to influential outliers than least squares. See here for a dramatic example in the case of the Kendall.

$\endgroup$
  • $\begingroup$ (+1) Great explanation! Is there any reason why Kendall seems to be more preferred to Spearman in this context (at least judging from the fact that Kendall correlation corresponds to a slope estimator that has a name, Theil-Sen, whereas Spearman one does not)? $\endgroup$ – amoeba Jul 31 '14 at 10:45
  • 4
    $\begingroup$ There are a number of reasons why this seems to be the case. First is that the Theil-Sen line has a simply-described estimator (median of the pairwise slopes), which the Spearman lacks; in small samples it's very suitable for hand calculation. The Kendall correlation approaches normality faster and is more mathematically tractable. See also here and here. $\endgroup$ – Glen_b Jul 31 '14 at 11:35
20
$\begingroup$

The proportional odds (PO) model generalizes Wilcoxon and Kruskal-Wallis tests. Spearman's correlation when $X$ is binary is the Wilcoxon test statistic simply translated. So you could say that the PO model is a unifying method. Since the PO model can have as many intercepts as there are unique values of $Y$ (less one), it handles both ordinal and continuous $Y$.

The numerator of the score $\chi^2$ statistic in the PO model is exactly the Wilcoxon statistic.

The PO model is a special case of a more general family of cumulative probability (some call cumulative link) models including the probit, proportional hazards, and complementary log-log models. For a case study see Chapter 15 of my Handouts.

$\endgroup$
4
$\begingroup$

Aaron Han (1987 in econometrics) proposed the Maximum Rank Correlation estimator that fits regression models by maximizing tau. Dougherty and Thomas (2012 in the psychology literature) recently proposed a very similar algorithm. There is an abundance of work on the MRC illustrating its properties.

Aaron K. Han, Non-parametric analysis of a generalized regression model: The maximum rank correlation estimator, Journal of Econometrics, Volume 35, Issues 2–3, July 1987, Pages 303-316, ISSN 0304-4076, http://dx.doi.org/10.1016/0304-4076(87)90030-3. (http://www.sciencedirect.com/science/article/pii/0304407687900303)

Dougherty, M. R., & Thomas, R. P. (2012). Robust decision making in a nonlinear world. Psychological review, 119 (2), 321. Retrieved from http://damlab.umd.edu/pdf%20articles/DoughertyThomas2012Rev.pdf.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.