4
$\begingroup$

I'm interested in assessing model performance on data with an ordinal categorical dependent variable. For my use case, the ideal metric would:

  1. Not assume equal intervals between classes or that recoding to a continuous scale is appropriate
  2. Be scale independent
  3. Give preference to models that rank the outcomes accurately, with higher penalties for mis-ranking classes with a larger degree of difference (e.g., Excellent > Poor > Good is better than Excellent > Very Poor > Good)
  4. Accept continuous predictions and be indifferent to their distributions

For example, suppose we have the following test set, where "response" is 5-category ordinal response and "pred1", "pred2", and "pred3" are predictions:

id      response   pred1    pred2    pred3
 1     Excellent    1.00      150       10
 2          Good     .80       39        9
 3          Good     .85       12        5
 4          Fair     .40       11        4
 5          Poor     .39       10        3
 6     Very Poor     .20        3        2
 .             .       .        .        .
 .             .       .        .        .

For my purposes, the ideal metric would score all three predictions as equally accurate since all three perfectly rank the response.

What are my options and the benefits/drawbacks to each? Bonus points for references to R packages or functions.

$\endgroup$
11
$\begingroup$

A good measure is Somers' Dxy rank correlation, a generalization of ROC area for ordinal or continuous Y. It is computed for ordinal proportional odds regression in the lrm function in the rms package.

$\endgroup$
  • $\begingroup$ This is exactly what I was looking for – I just didn't realize Somers generalized to ordinal Y. (I've used your somers2() function from the HMisc package, but it only accepts binary Y.) Thank you! $\endgroup$ – danpelota Apr 20 '11 at 15:33
  • $\begingroup$ Franck, did you note my question on ordinal regression, on the outcome of various function from your rms package? How do I interpret a given value of Dxy (your answer here helped me, but can you tell more)? and a cross-validate Dxy? If you don’t have time to write a long answer, a bibliographic pointer would be welcome... $\endgroup$ – Elvis Feb 21 '12 at 21:06
  • $\begingroup$ I just put an answer on that page. Best not to create 2 pages. $\endgroup$ – Frank Harrell Feb 22 '12 at 3:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.