Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I have data where a rater predicts quality on a continuous scale, and quality is then validated in buckets. For example, one rater might give scores of 0.2, 0.4, 0.7, and 0.6 for cases with outcomes of LOW, MEDIUM, HIGH, LOW (respectively). This would be an overall decent rater aside from the fact that the fourth case received a higher rating than the second, yet experienced a worse outcome.

I'm looking for a way to compute a precision metric for this type of rating problem. Hand & Till's multi-outcome AUC (R package) measure is helpful context, but insufficient since the raters aren't predicting each individual outcome. Rather, I'm looking to leverage the ordinal nature of the data.

share|improve this question
    
Does "L, M, H, L" mean "low, medium, high, low"? –  Stephan Kolassa Jan 15 '13 at 7:22
    
Yes, updated for clarity –  Max Ghenis Jan 15 '13 at 7:39
    
How about looking at ordinal or rank correlations (Spearman or Kendall)? –  Stephan Kolassa Jan 15 '13 at 7:43
    
Intriguing, I'll look into it. Seems a bit odd to resort to ranking the output for only three levels, but will definitely give it a try. I suppose I was hoping there might be something more similar to AUC. For example, picking a random pair of points with different outcomes and seeing the % time that the higher-ranked one is rated higher. –  Max Ghenis Jan 15 '13 at 8:27

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.