I am trying to understand the relative behavior of the following rank correlation statistics:

  1. Spearman coefficient
  2. Kendall Tau / Concordance percentage
  3. Normalized Gini coefficient (area under curve of percentage captured versus percentage observations)
  4. Normalized Area under ROC curve (for binary classifiers)

I don't believe any of these are functionally related to the others. The accepted answer here references this paper and Spearman and Kendall are highly correlated (as one would expect).

Are there good intuitions behind/discussions of relative (across datasets) or absolute (for a given dataset) differences for (any pair of) these measures?

  • $\begingroup$ ROC curve is not a rank order correlation statistic as far as I'm aware, I don't believe the Gini coefficient is either, as it is a measure of statistical dispersion. Perhaps I'm wrong or have misinterpreted. $\endgroup$ – analystic Oct 1 '12 at 4:46
  • $\begingroup$ Sorry, I did a poor job of defining what I mean -- rank by the prediction, not by the target. Then the Gini statistic (similarly ROC) as intended is similarly to how the term is used here. This is a somewhat standard usage (see for example, this random paper (pdf) from near the top of google's results. $\endgroup$ – cohoz Oct 2 '12 at 1:44
  • 1
    $\begingroup$ I think you're misunderstanding two uses of the term "rank". Rho and Tau refer to calculating a monotonic relationship between two variables. The other two metrics are used for determining the effectiveness of a statistical model. In this sense, they may "rank" models according to their predictive power, and provide information about the models, however they are not measures of rank correlation. $\endgroup$ – analystic Oct 2 '12 at 11:43
  • $\begingroup$ Can you explain further? I am trying to see how well a predicted score ranks a target variable. The Gini is one way to do this. Why isn't Spearman's rho another? $\endgroup$ – cohoz Oct 3 '12 at 1:20
  • $\begingroup$ how does a "predicted scored rank a target variable"? I don't know what you mean by this. You have target variables and you have models which predict target variables, they don't rank them. However, the Gini index is not a model and doesn't predict a dependent variable, as I understand its use. $\endgroup$ – analystic Oct 5 '12 at 10:07

The proposed question is rather complicated. As analystic already pointed out, I don't think all these measures can be compared straightforwardly, because rank correlation coefficients, Gini coefficient, and AUC (area under ROC curve) are generally defined on different domains.

However, there is a very close relation between Kendall's $\tau$ and Spearman's $\rho$, the two rank correlation coefficients in the list. While the paper cohoz mentioned has demonstrated their relation empirically (Figure 3), this relation can actually be quantified theoretically. Let $\pi$ and $\sigma$ be two rankings, and $\pi(i)$ and $\sigma(i)$ be the ranks of item $i$ in $\pi$ and $\sigma$, respectively. The Kendall distance and Spearman distance between $\pi$ and $\sigma$ are defined as follows: $$ K(\pi,\sigma) = \# \lbrace \; (i,j) \, \vert \, \pi(i)>\pi(j) \text{ and } \sigma(i)<\sigma(j) \; \rbrace $$ $$ S(\pi,\sigma) = \sum_i \left( \pi(i) - \sigma(i)\right)^2 $$ We have the following relation between $K$ and $S$ following [Diaconis and Graham 1977]: $$ \frac{1}{\sqrt{n}}K(\pi,\sigma) \le S(\pi,\sigma) \le 2K(\pi,\sigma) $$ Because the rank correlation coefficients are just the normalization of the rank distances to the interval $[-1,1]$, similar inequalities can be easily derived between $\tau$ and $\rho$. In the statistical ranking literature, results are mostly represented in terms of distances rather than coefficients.

Two more things:

  1. The rankings $\pi$ and $\sigma$ must be complete rankings in order to make this inequality hold. That is, they cannot be partial rankings.
  2. In case one is interested in $\tau$ and $\rho$ defined not only on rankings but on continuous random variables, the situation is more involved. Here is a related paper by Fredicks and Nelsen.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.