# Scoring predictions of an ordinal variable

I read about using scoring rules to evaluate the performance of predictive models. In the Wikipedia article about the Brier score, it is stated:

The Brier score is appropriate for binary and categorical outcomes that can be structured as true or false, but is inappropriate for ordinal variables which can take on three or more values (this is because the Brier score assumes that all possible outcomes are equivalently "distant" from one another).

What are some examples of scoring rules used for evaluating predictions of ordinal variables? For example, if the set of possible outcomes is $\{0, 1, 2, 3, 4, 5, 6, 7, 8, 9\}$ and a predictive model yields $p_6 = 0.3$, $p_7 = 0.5$, $p_8 = 0.2$ (where $p_i$ is the believed probability that the outcome will be $i$), this predictive model should receive a lower score if the outcome is $3$ than if it is $4$.

The Brier score is a wonderful scoring rule for binary $Y$ and there are decompositions of it into discrimination and calibration components. But as you nicely stated it doesn't extend to ordinal $Y$. Although it does not combine calibration + discrimination, Somers' $D_{xy}$ rank correlation between the linear predictor and $Y$ is an excellent measure of pure discrimination for ordinal or continuous $Y$. For the many-category case, Spearman's $\rho$ is also worth considering.

• The problem that I am up against, and what makes proper scoring rules appealing, is that I need to score predictions even when I only have a single data point for the outcome variable. (This is because my predictive model produces many different distributions during its run, and many of these will only be seen once - therefore tested against a single data point.) Don't Somers' $D_{xy}$ and Spearman's $\rho$ require a larger set to be appropriate? Jul 20, 2013 at 21:42
• Yes, at least two points. I understand the need to predict as you go. Why the need to accuracy score point-by-point? Jul 20, 2013 at 21:51
• I simply don't have more than a single point for many of the predictions. I don't want a model to be unfairly penalized because it never served the same prediction more than once (which I assume using a statistic that prefers a larger sample size may do). Jul 21, 2013 at 6:00
• I still don't have any clear picture of what you are trying to accomplish by computing an accuracy score for each new prediction and observation of $Y$. Are you trying to update model parameters as you go? Are you trying to find a new accuracy measure? The fact that some accuracy measures (e.g., $c$-index = generalized ROC area, linearly related to $D_{xy}$) require more than one observation to compute implies in no way that the index somehow penalizes predictions. Jul 21, 2013 at 11:27
• @FrankHarrell: I know I'm late, but what the OP wants to do is likely to evaluate an entire prediction system. Multiple instances may be predicted, each with its own setting to all predictors, and therefore with different predicted probabilities, and each instance of course with only one actual observation. This is quite common, and proper scoring rules like the RPS provide a nice framework, where one would calculate and average the RPS over multiple such instances. Jul 26 at 8:52

This is exactly what the ranked probability score (RPS) does.

Assume your predicted cumulative probabilities of $$r$$ possible outcomes are $$P_1, \dots, P_r$$, i.e., $$P_i$$ is the predicted probability that the outcome is less than or equal to $$i$$ (as per the order in your ordinal outcome). Then the RPS for an observed outcome $$y$$ is calculated as $$\text{RPS} = \sum_{i=1}^r(P_i-1_{y\leq i})^2$$ (Gneiting & Katzfuss, 2014, who give the continuous version, i.e., the CRPS).

Note that this uses the "negative" convention in scoring rules, i.e., that "smaller is better". The original question prefers the opposite "positive" convention, which is a little less common in the literature. Feel free to use a minus sign if the orientation is important to you. (And whatever you prefer, just note explicitly which convention you are following.)

The RPS is a proper scoring rule and correctly yields a better (i.e., lower) score for an observed outcome of 4 than for 3 in your example, since 4 is closer to the predicted probability mass. In R:

# predicted probabilities are assumed to be ordered
# from "smallest" to "largest" outcome

rps <- function(predicted_probabilities, outcome) {
result <- 0
for ( ii in seq_along(predicted_probabilities) ) {
result <- result + (cumsum(predicted_probabilities)[ii]-(outcome<=ii))^2
}
result
}

predicted_probabilities <- structure(c(0,0,0,0,0,0,0.3,0.5,0.2,0),.Names=0:9)
rps(predicted_probabilities,3)  # yields 4.53
rps(predicted_probabilities,4)  # yields 3.53


Two observations in the spirit of Why is LogLoss preferred over other proper scoring rules?:

First, the RPS is of course non-local, in the sense that two predicted probability masses will yield different RPSs for the same outcome even if the predictions for the observed outcome are the same:

rps(c(0.4,0.3,0.3),1)   # yields 0.45
rps(c(0.4,0.4,0.2),1)   # yields 0.4


This of course makes perfect sense in the specific context of ordinal outcomes we want to predict.

Second, the RPS is not additive. Suppose we add a new outcome to our prediction, with a predicted probability of zero, but observe the same outcome as before. If the "new" possible outcome is either the lowest or the largest outcome after the addition, the RPS will not change:

rps(c(0.4,0.3,0.3),1)   # yields 0.45
rps(c(0.4,0.3,0.3,0),1) # yields 0.45
rps(c(0,0.4,0.3,0.3,0),2)   # yields 0.45


However, if we add the new outcome "in the middle", the RPS does indeed change:

rps(c(0.4,0.3,0.3),1)   # yields 0.45
rps(c(0.4,0.3,0,0.3),1) # yields 0.54


This may or may not make sense in your specific application, but it is something to keep in mind.

• Nice to know about this. It is a summed Brier score multiplied by the sample size. Jul 26 at 11:27