Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 1 down vote favorite

I have data where a rater predicts quality on a continuous scale, and quality is then validated in buckets. For example, one rater might give scores of 0.2, 0.4, 0.7, and 0.6 for cases with outcomes of LOW, MEDIUM, HIGH, LOW (respectively). This would be an overall decent rater aside from the fact that the fourth case received a higher rating than the second, yet experienced a worse outcome.

I'm looking for a way to compute a precision metric for this type of rating problem. Hand & Till's multi-outcome AUC (R package) measure is helpful context, but insufficient since the raters aren't predicting each individual outcome. Rather, I'm looking to leverage the ordinal nature of the data.

share|improve this question
Does "L, M, H, L" mean "low, medium, high, low"? – Stephan Kolassa Jan 15 at 7:22
Yes, updated for clarity – Max Ghenis Jan 15 at 7:39
How about looking at ordinal or rank correlations (Spearman or Kendall)? – Stephan Kolassa Jan 15 at 7:43
Intriguing, I'll look into it. Seems a bit odd to resort to ranking the output for only three levels, but will definitely give it a try. I suppose I was hoping there might be something more similar to AUC. For example, picking a random pair of points with different outcomes and seeing the % time that the higher-ranked one is rated higher. – Max Ghenis Jan 15 at 8:27

Know someone who can answer? Share a link to this question via email, Google+, Twitter, or Facebook.

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.