3
$\begingroup$

Somers' D as defined for example in "The predictive accuracy of credit ratings: Measurement and statistical inference" by Walter Orth is defined for the case when Y is predicted by X as $$ D_{XY} = \frac{1}{n_u} \sum_{i=1}^n\sum_{j>i} c_{ij}, $$ where $c_{ij}$ counts 1 for concordant pairs of $(X,Y)$ and -1 for discordant pairs and 0 else. So far it all rather simple. The denominator $n_u$ is defined as $$ n_u = \sum_{i=1}^n\sum_{j>i} 1_{ \{Y_i \neq Y_j \}}. $$ Thus it counts all pairs but the ones where the Y-values are equal. Thus ties in $X$ are counted but ties in $Y$ reduce the denominator and therefore increase $D_{XY}$.

How does this carry over to the case of validation of a model? Can we just set $X = \hat{Y}$ by which I mean the model prediction and $Y$ are the realized values? Then I look at concordant and discordant pars of the prediction $\hat{Y}$ and the realization. For the denominator I don't count realization pairs $i,j$ with $Y_i = Y_j$. The author says that in such case one can not judge the performance of the prediction.

If $Y$ is a default indicator of a credit scoring model, then I would interpret $X$ as the probability of default or a risk class from my model and $Y$ the default observation. In the paper linked it is mentioned that in this case the above gives the accuracy ratio (= Gini coeffient).

My question: do you agree that a useful interpretation of Somers' D is as above: $X$ is the prediction and $Y$ the observed realization with the stated consequences? Especially, on this case ties of predictions are counted in the denominator but ties in the relization not.

$\endgroup$
2
+50
$\begingroup$

Somers' $D_{xy}$ is a general measure of predictive discrimination which can handle binary, ordinal, and censored time-to-event outcome variables. In the binary $Y$ case it is just $2 (c - 0.5)$ where $c$ is the concordance probability, AKA area under the ROC curve. It is a very interpretable measure of predictive discrimination but like AUROC is not sensitive enough for choosing or comparing models.

To your question, yes we compute $D_{xy}$ by playing $\hat{Y}$ against $Y$ and using the ordinary Somers' rank correlation formula you have listed, where ties on $Y$ are ignored (not penalized against). The only difference is that the estimate of $D$ will be biased when you have overfit the data patterns. For that reason we don't concentrate on the "apparent $D_{xy}$" but rather on the overfitting-corrected version of it. For most cases this is most efficiently estimated using the bootstrap. Alternatively, 100 repeats of 10-fold cross-validation may be used.

Details are in Regression Modeling Strategies book and course notes. Start with the chapter on binary logistic regression where I cover $D_{xy}$. The R rms package validate.* functions compute apparent and validated (overfitting-corrected) $D_{xy}$.

The "not sensitive enough" comment relates to rank measures not giving a sufficient reward for extreme predictions that are "right". Sensitive measures are discussed here.

$\endgroup$
  • $\begingroup$ Thank you for your answer. Seeing it it reminds me another question: I was curious for a long time does Sommers D in NONBINARY case has geometric interpretation like area under surface or similar? Might not be good place to write it under other person question, i can write a separate question if threre some answer. $\endgroup$ – Alexander Chervov Jun 19 at 14:31
  • 1
    $\begingroup$ In the general case, where $Y$ is at least ordinal, it is still the difference between concordance and discordance probabilities, but you don't have an ROC curve interpretation. You can interpret its concordance probability by showing a scatterplot and envisioning the proportion of all possible connections between two points that have a positive slope. $\endgroup$ – Frank Harrell Jun 19 at 16:37
  • $\begingroup$ Thank you! Is there any reference for 'proportion of all possible...'? $\endgroup$ – Alexander Chervov Jun 19 at 18:21
  • $\begingroup$ Best idea for now is to look at an intro nonparametric statistics text where the Wilcoxon test is introduced (or at wikipedia). The Mann-Whitney U-statistic is the concordance probability. $\endgroup$ – Frank Harrell Jun 20 at 2:40
  • 1
    $\begingroup$ Yes. You can use different rank correlation indexes if you want to penalize differently for ties, e.g. Goodman-Kruskal $\gamma$ discards ties both on $X$ and $Y$. $\endgroup$ – Frank Harrell Jun 20 at 17:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.