I have data where a rater predicts quality on a continuous scale, and quality is then validated in buckets. For example, one rater might give scores of 0.2, 0.4, 0.7, and 0.6 for cases with outcomes of LOW, MEDIUM, HIGH, LOW (respectively). This would be an overall decent rater aside from the fact that the fourth case received a higher rating than the second, yet experienced a worse outcome.

I'm looking for a way to compute a precision metric for this type of rating problem. Hand & Till's multi-outcome AUC (R package) measure is helpful context, but insufficient since the raters aren't predicting each individual outcome. Rather, I'm looking to leverage the ordinal nature of the data.

  • $\begingroup$ Does "L, M, H, L" mean "low, medium, high, low"? $\endgroup$ – Stephan Kolassa Jan 15 '13 at 7:22
  • $\begingroup$ Yes, updated for clarity $\endgroup$ – Max Ghenis Jan 15 '13 at 7:39
  • 1
    $\begingroup$ How about looking at ordinal or rank correlations (Spearman or Kendall)? $\endgroup$ – Stephan Kolassa Jan 15 '13 at 7:43
  • $\begingroup$ Intriguing, I'll look into it. Seems a bit odd to resort to ranking the output for only three levels, but will definitely give it a try. I suppose I was hoping there might be something more similar to AUC. For example, picking a random pair of points with different outcomes and seeing the % time that the higher-ranked one is rated higher. $\endgroup$ – Max Ghenis Jan 15 '13 at 8:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.