Scoring predictions of an ordinal variable

Question

I read about using scoring rules to evaluate the performance of predictive models. In the Wikipedia article about the Brier score, it is stated:

The Brier score is appropriate for binary and categorical outcomes that can be structured as true or false, but is inappropriate for ordinal variables which can take on three or more values (this is because the Brier score assumes that all possible outcomes are equivalently "distant" from one another).

What are some examples of scoring rules used for evaluating predictions of ordinal variables? For example, if the set of possible outcomes is $\{0, 1, 2, 3, 4, 5, 6, 7, 8, 9\}$ and a predictive model yields $p_6 = 0.3$, $p_7 = 0.5$, $p_8 = 0.2$ (where $p_i$ is the believed probability that the outcome will be $i$), this predictive model should receive a lower score if the outcome is $3$ than if it is $4$.

Answer 1

The Brier score is a wonderful scoring rule for binary $Y$ and there are decompositions of it into discrimination and calibration components. But as you nicely stated it doesn't extend to ordinal $Y$. Although it does not combine calibration + discrimination, Somers' $D_{xy}$ rank correlation between the linear predictor and $Y$ is an excellent measure of pure discrimination for ordinal or continuous $Y$. For the many-category case, Spearman's $\rho$ is also worth considering.

Answer 2

This is exactly what the ranked probability score (RPS) does.

Assume your predicted cumulative probabilities of $r$ possible outcomes are $P_1, \dots, P_r$, i.e., $P_i$ is the predicted probability that the outcome is less than or equal to $i$ (as per the order in your ordinal outcome). Then the RPS for an observed outcome $y$ is calculated as $$ \text{RPS} = \sum_{i=1}^r(P_i-1_{y\leq i})^2 $$ (Gneiting & Katzfuss, 2014, who give the continuous version, i.e., the CRPS).

Note that this uses the "negative" convention in scoring rules, i.e., that "smaller is better". The original question prefers the opposite "positive" convention, which is a little less common in the literature. Feel free to use a minus sign if the orientation is important to you. (And whatever you prefer, just note explicitly which convention you are following.)

The RPS is a proper scoring rule and correctly yields a better (i.e., lower) score for an observed outcome of 4 than for 3 in your example, since 4 is closer to the predicted probability mass. In R:

# predicted probabilities are assumed to be ordered
# from "smallest" to "largest" outcome

rps <- function(predicted_probabilities, outcome) {
    result <- 0
    for ( ii in seq_along(predicted_probabilities) ) {
        result <- result + (cumsum(predicted_probabilities)[ii]-(outcome<=ii))^2
    }
    result
}

predicted_probabilities <- structure(c(0,0,0,0,0,0,0.3,0.5,0.2,0),.Names=0:9)
rps(predicted_probabilities,3)  # yields 4.53
rps(predicted_probabilities,4)  # yields 3.53

---

Two observations in the spirit of Why is LogLoss preferred over other proper scoring rules?:

First, the RPS is of course non-local, in the sense that two predicted probability masses will yield different RPSs for the same outcome even if the predictions for the observed outcome are the same:

rps(c(0.4,0.3,0.3),1)   # yields 0.45
rps(c(0.4,0.4,0.2),1)   # yields 0.4

This of course makes perfect sense in the specific context of ordinal outcomes we want to predict.

Second, the RPS is not additive. Suppose we add a new outcome to our prediction, with a predicted probability of zero, but observe the same outcome as before. If the "new" possible outcome is either the lowest or the largest outcome after the addition, the RPS will not change:

rps(c(0.4,0.3,0.3),1)   # yields 0.45
rps(c(0.4,0.3,0.3,0),1) # yields 0.45
rps(c(0,0.4,0.3,0.3,0),2)   # yields 0.45

However, if we add the new outcome "in the middle", the RPS does indeed change:

rps(c(0.4,0.3,0.3),1)   # yields 0.45
rps(c(0.4,0.3,0,0.3),1) # yields 0.54

This may or may not make sense in your specific application, but it is something to keep in mind.