I have a model built to predict an ordinal response with 3 levels (win, draw or lose, say) and I would like to evaluate predictive accuracy.

Is it appropriate to use the Brier Score:

$$ \sum_{i=1}^{R}\left (f_{i} - o_{i} \right )^{2} $$

where $R$ is the number of response levels, $f$ is the predicted probability and $o$ is the observed outcome - which is 1 for the value of $i$ for which the event occurred - in order to determine the accuracy of these probability predictions?

Does the ordinality of the data break any of the assumptions made by the RPS? It is briefly mentioned, without explanation, in the wikipedia that this is not suitable for ordinal variables:

The Brier score is appropriate for binary and categorical outcomes that can be structured as true or false, but is inappropriate for ordinal variables which can take on three or more values.

EDIT: Going with the win, draw or lose example; lets say we predicted team A had a 0.5 chance of winning, team B had a 0.3 chance of winning and that there was a 0.2 chance of the draw. The actual result of the match was a win for team A, so the corresponding Brier score would be:

$$ \left(0.5 - 1 \right )^{2} + \left(0.3 - 0 \right )^{2} + \left(0.2 - 0 \right )^{2} = 0.255 $$

where the observed outcome is 1 for the win and 0 for the draw and loss.


marked as duplicate by kjetil b halvorsen, mdewey, Peter Flom Jan 8 at 11:23

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 1
    $\begingroup$ Hint: list some examples of the quantity that you would be squaring. $\endgroup$ – rolando2 Jan 7 at 21:38
  • $\begingroup$ Other dups: stats.stackexchange.com/questions/220081/… $\endgroup$ – kjetil b halvorsen Jan 8 at 9:41
  • 1
    $\begingroup$ @kjetilbhalvorsen you're spot on, those other questions contained the information I required. Thanks. $\endgroup$ – Michael Pyle Jan 8 at 16:41