I have data where a rater predicts quality on a continuous scale, and quality is then validated in buckets. For example, one rater might give scores of 0.2, 0.4, 0.7, and 0.6 for cases with outcomes of LOW, MEDIUM, HIGH, LOW (respectively). This would be an overall decent rater aside from the fact that the fourth case received a higher rating than the second, yet experienced a worse outcome.
I'm looking for a way to compute a precision metric for this type of rating problem. Hand & Till's multi-outcome AUC (R package) measure is helpful context, but insufficient since the raters aren't predicting each individual outcome. Rather, I'm looking to leverage the ordinal nature of the data.