Model performance metrics for ordinal response

I'm interested in assessing model performance on data with an ordinal categorical dependent variable. For my use case, the ideal metric would:

1) Not assume equal intervals between classes or that recoding to a continuous scale is appropriate
2) Be scale independent
3) Give preference to models that rank the outcomes accurately, with higher penalties for mis-ranking classes with a larger degree of difference (e.g., Excellent > Poor > Good is better than Excellent > Very Poor > Good)
4) Accept continuous predictions and be indifferent to their distributions

For example, suppose we have the following test set, where "response" is 5-category ordinal response and "pred1", "pred2", and "pred3" are predictions:

id      response   pred1    pred2    pred3
1     Excellent    1.00      150       10
2          Good     .80       39        9
3          Good     .85       12        5
4          Fair     .40       11        4
5          Poor     .39       10        3
6     Very Poor     .20        3        2
.             .       .        .        .
.             .       .        .        .


For my purposes, the ideal metric would score all three predictions as equally accurate since all three perfectly rank the response.

What are my options and the benefits/drawbacks to each? Bonus points for references to R packages or functions. Thanks!

-

 This is exactly what I was looking for – I just didn't realize Somers generalized to ordinal Y. (I've used your somers2() function from the HMisc package, but it only accepts binary Y.) Thank you! – danpelota Apr 20 '11 at 15:33 Franck, did you note my question on ordinal regression, on the outcome of various function from your rms package? How do I interpret a given value of Dxy (your answer here helped me, but can you tell more)? and a cross-validate Dxy? If you don’t have time to write a long answer, a bibliographic pointer would be welcome... – Elvis Feb 21 '12 at 21:06 I just put an answer on that page. Best not to create 2 pages. – Frank Harrell Feb 22 '12 at 3:03