12
$\begingroup$

I'm using 2 kind of logistic regression - one is the simple type, for binary classification, and the other is ordinal logistic regression. For calculating the accuracy of the first, I used cross-validation, where I computed the AUC for each fold and than calculated the mean AUC. How can I do it for the ordinal logistic regression? I've heard about generalized ROC for multi-class predictors, but I'm not sure how to compute it.

Thanks!

Improve this question
$\endgroup$
1

2 Answers 2

Reset to default
9
$\begingroup$

I only like the area under the ROC curve ($c$-index) because it happens to be a concordance probability. $c$ is a building block of rank correlation coefficients. For example, Somers' $D_{xy} = 2\times (c - \frac{1}{2})$. For ordinal $Y$, $D_{xy}$ is an excellent measure of predictive discrimination, and the R rms package provides easy ways to get bootstrap overfitting-corrected estimates of $D_{xy}$. You can backsolve for a generalized $c$-index (generalized AUROC). There are reasons not to consider each level of $Y$ separately because this does not exploit the ordinal nature of $Y$.

In rms there are two functions for ordinal regression: lrm and orm, the latter handling continuous $Y$ and providing more distribution families (link functions) than proportional odds.

Improve this answer
$\endgroup$
4
  • $\begingroup$ The main issue will be how does rms calculate the $c-index$ used in Sommer's $D_{xy}$? $\endgroup$
    – Chamberlain Mbah
    Commented Dec 10, 2013 at 13:48
  • 1
    $\begingroup$ It is spelled Somer's. The generalized $c$-index is simply computed by backsolving the equation I listed above. Internal, all possible combinations of observations having different $Y$ values are examined, and the fraction of such pairs for which predictions are in the same order is the estimate of the concordance probability. I misstated one thing: The orm function uses Spearman's $\rho$ instead of $D_{xy}$. $\endgroup$
    – Frank Harrell
    Commented Dec 10, 2013 at 14:56
  • $\begingroup$ Thanks for the spelling correction. In ordinal regression it will be much more interesting not only to look at pairwise ordering as is done in orm function you mentioned, but also look at consistent ordering ( with ternary or higher operators) depending on the number of classes you have. In summary what I am saying is: with a cumulative logistic regression fitted for example, the ordering of classes is taken care of in the model. A predictive measure should also be able not to do pairwise comparison $P(pred_1<pred_2|obs_1<obs_2)$ but comparison of the form $P(pred_1<pred_2<pred_3|obs_1<obs_2<o $\endgroup$
    – Chamberlain Mbah
    Commented Dec 10, 2013 at 16:12
  • $\begingroup$ Not knowing about such measures my first reaction is that they are setting a high bar to hurdle. $\endgroup$
    – Frank Harrell
    Commented Dec 10, 2013 at 20:28
3
$\begingroup$

AUC for ordinal regression is something tricky. You might want to calculate the AUC for each class by creating dummies to take value 1 for the class you are calculating the AUC and 0 for the rest of the other classes. If you have 4 classes then you will create 4 AUCs and plot them on the same graph. The main problem with this method is the fact that it penalizes miss-classification equally. Much more intuitively miss-classifying a class 1 into class 3 should be worst than miss-classifying class 1 into class 2.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.