# Somers' D for model validation

Somers' D as defined for example in "The predictive accuracy of credit ratings: Measurement and statistical inference" by Walter Orth is defined for the case when Y is predicted by X as $$D_{XY} = \frac{1}{n_u} \sum_{i=1}^n\sum_{j>i} c_{ij},$$ where $$c_{ij}$$ counts 1 for concordant pairs of $$(X,Y)$$ and -1 for discordant pairs and 0 else. So far it all rather simple. The denominator $$n_u$$ is defined as $$n_u = \sum_{i=1}^n\sum_{j>i} 1_{ \{Y_i \neq Y_j \}}.$$ Thus it counts all pairs but the ones where the Y-values are equal. Thus ties in $$X$$ are counted but ties in $$Y$$ reduce the denominator and therefore increase $$D_{XY}$$.

How does this carry over to the case of validation of a model? Can we just set $$X = \hat{Y}$$ by which I mean the model prediction and $$Y$$ are the realized values? Then I look at concordant and discordant pars of the prediction $$\hat{Y}$$ and the realization. For the denominator I don't count realization pairs $$i,j$$ with $$Y_i = Y_j$$. The author says that in such case one can not judge the performance of the prediction.

If $$Y$$ is a default indicator of a credit scoring model, then I would interpret $$X$$ as the probability of default or a risk class from my model and $$Y$$ the default observation. In the paper linked it is mentioned that in this case the above gives the accuracy ratio (= Gini coeffient).

My question: do you agree that a useful interpretation of Somers' D is as above: $$X$$ is the prediction and $$Y$$ the observed realization with the stated consequences? Especially, on this case ties of predictions are counted in the denominator but ties in the relization not.

Somers' $$D_{xy}$$ is a general measure of predictive discrimination which can handle binary, ordinal, and censored time-to-event outcome variables. In the binary $$Y$$ case it is just $$2 (c - 0.5)$$ where $$c$$ is the concordance probability, AKA area under the ROC curve. It is a very interpretable measure of predictive discrimination but like AUROC is not sensitive enough for choosing or comparing models.

To your question, yes we compute $$D_{xy}$$ by playing $$\hat{Y}$$ against $$Y$$ and using the ordinary Somers' rank correlation formula you have listed, where ties on $$Y$$ are ignored (not penalized against). The only difference is that the estimate of $$D$$ will be biased when you have overfit the data patterns. For that reason we don't concentrate on the "apparent $$D_{xy}$$" but rather on the overfitting-corrected version of it. For most cases this is most efficiently estimated using the bootstrap. Alternatively, 100 repeats of 10-fold cross-validation may be used.

Details are in Regression Modeling Strategies book and course notes. Start with the chapter on binary logistic regression where I cover $$D_{xy}$$. The R rms package validate.* functions compute apparent and validated (overfitting-corrected) $$D_{xy}$$.

The "not sensitive enough" comment relates to rank measures not giving a sufficient reward for extreme predictions that are "right". Sensitive measures are discussed here.

• Thank you for your answer. Seeing it it reminds me another question: I was curious for a long time does Sommers D in NONBINARY case has geometric interpretation like area under surface or similar? Might not be good place to write it under other person question, i can write a separate question if threre some answer. – Alexander Chervov Jun 19 at 14:31
• In the general case, where $Y$ is at least ordinal, it is still the difference between concordance and discordance probabilities, but you don't have an ROC curve interpretation. You can interpret its concordance probability by showing a scatterplot and envisioning the proportion of all possible connections between two points that have a positive slope. – Frank Harrell Jun 19 at 16:37
• Thank you! Is there any reference for 'proportion of all possible...'? – Alexander Chervov Jun 19 at 18:21
• Best idea for now is to look at an intro nonparametric statistics text where the Wilcoxon test is introduced (or at wikipedia). The Mann-Whitney U-statistic is the concordance probability. – Frank Harrell Jun 20 at 2:40
• Yes. You can use different rank correlation indexes if you want to penalize differently for ties, e.g. Goodman-Kruskal $\gamma$ discards ties both on $X$ and $Y$. – Frank Harrell Jun 20 at 17:34