# Alternative notions to that of proper scoring rules, and using scoring rules to evaluate models

A scoring rule is a means of evaluating an agent's guess of the probabilities associated with a categorical event, given a (categorical) outcome of the event. Depending on the guess and the observed outcome, the scoring rule gives the agent a score (a real number). A scoring rule is supposed to assign scores such that, on average, the agent with the least score makes the most accurate guesses. (Conventions differ as to whether scoring rules are framed in terms of minimization or maximization. Here I'm taking the minimization view.)

An important property of scoring rules is whether they're a proper scoring rule; that is, whether they give the least mean score when an agent guesses the true probabilities (or, in a subjective Bayesian framing, they give the least posterior mean score, given the agent's own priors, when an agent uses its own degrees of belief as its guesses). In the case of a binary event, squared error from 0 or 1 (the Brier score) is a proper scoring rule whereas absolute error is not. Why? Well, the criterion of properness is based on the mean, and the mean is the measure of central tendency that minimizes the sum of squared differences, but need not minimize absolute error.

This line of thinking suggests that if we replace the mean in the definition of a proper scoring rule with some other statistical functional, such as the median, then we'll get an analogous sort of rich family of proper scoring rules. It's not unreasonable to imagine a situation where an agent wants to minimize its median score rather than its mean score. Actually, it seems that there are no nontrivial median-proper scoring rules. Considering the case of a binary event again, if the true probability is less than 1/2, then the median score of an agent will be equal to whatever score is given to the agent when the event doesn't occur, regardless of the event's exact probability. Analogous shenanigans seem to occur if we replace the mean by, say, the geometric mean.

So, is there a sense that in order for the theory of proper scoring rules to work as intended, the statistical functional must be the mean?

I realize this is a vague question, and the best answer is likely to be an explanation of why the question doesn't really make sense, so here's the context where I find myself asking it, to help you un-confuse me. I'm a psychologist of decision-making, and I often find myself wanting to quantify the performance (either predictive performance, under cross-validation, or model fit post-hoc) of a model that spits out probabilities of what people will choose in a binary-decision scenario. The above discussion suggests I should use a proper scoring rule. Annoyingly, proper scoring rules aren't on the same scale as probabilities. I find myself wanting to, for example, take the square root of the mean squared error rather than just looking at the mean squared error (that is, the mean Brier score), but in the case of one trial, the RMSE is equivalent to absolute error, which isn't proper, so wouldn't I then think that models which are less accurate are better? Evidently I can't just change my method of evaluating scoring rules from one in terms of means to one in terms of, e.g., medians. Must I simply familiarize myself with the scale of one of the usual proper scoring rules, or use a signal-detection statistic like area under the ROC curve or d'?

An additional complication is that for one study I'm looking at parametrically bootstrapped model fits, in accordance with Wagenmakers, Ratcliff, Gomez, and Iverson (2004), which means I'm looking at density plots of scores rather than individual scores. Then it's even less clear whether I should be concerned about properness or about some analogous criterion.

Edit: see this comment thread on Reddit for some more discussion.

Wagenmakers, E.-J., Ratcliff, R., Gomez, P., & Iverson, G. J. (2004). Assessing model mimicry using the parametric bootstrap. Journal of Mathematical Psychology, 48, 28–50. doi:10.1016/j.jmp.2003.11.004

• Am I right that you're asking two questions: 1 -- Can "proper" be re-defined in terms of the median score, rather than the expected score of a given forecast? 2 -- Are there proper scores for probabilities that are on the scale of the probability? – Fabian Apr 14 '15 at 7:37
• (1) I'm pretty sure the answer to that question is "no"; what I'm asking is if it makes sense to redefine "proper" in terms of anything other than the mean (i.e., expectation). (2) Yes, that's a question I'd like the answer to, but since the answer is again probably "no", I guess my follow-up would be "Then what's a good scoring rule that's interpretable in a way that relates naturally to probabilities?" – Kodiologist Apr 14 '15 at 12:48
• About (1), the following paper seems related to your question: ssc.upenn.edu/~fdiebold/papers/paper118/DieboldShin_SED.pdf The authors look at a case in which interest is not on the expected score, but on the distribution of scores. Interestingly, they again end up minimizing expected scores of a certain form (see Propositions 2.2 and 3.1). – Fabian Apr 14 '15 at 13:50
• Unfortunately, it seems that that paper is about forecasts of the same type as the DV, as opposed to this case where I'm asking about guesses of the probability of an event rather than a guess of the most probable event. The guesses are probabilities whereas the DV is realized in a discrete fashion. – Kodiologist Apr 14 '15 at 15:02

Contrary to what you said about geometric mean shenanigans, there are actually proper scoring rules for the geometric mean.

The geometric mean of a random variable $X$ is equal to $e^{E(\log X)}$. Therefore minimizing the geometric mean of a random score $S$ corresponds to minimizing the arithmetic mean of a random score $\log S$. So if $f(\hat p)$ is a standard proper scoring rule (where $f(\hat p)$ is the score you get if you predict a probability $\hat p$ and the event happens), then $g(\hat p) = \log f(\hat p)$ is a proper scoring rule for the geometric mean.

Similarly, the harmonic mean of $X$ is $E(X^{-1})^{-1}$, so $g(\hat p) = -f(\hat p)^{-1}$ is a harmonic-proper scoring rule. (The negative sign is in there so the coordinate transformation is monotone increasing.)

This works for any central tendency that is the arithmetic mean in a monotonically transformed space. The problem is that the median doesn't work like this. More generally, any central tendency with a nonzero breakdown point will not work, because it will be insensitive to changes in probability when $p$ is small. For instance the interquartile range won't work, because if $p < 0.25$, then the interquartile range of the scores doesn't depend on $p$ (so the same $\hat p$ must minimize the IQR for all values of $p$ less than $0.25$, which is bad).

Off the top of my head I can't think of any central tendencies with 0 breakdown point that can't be rewritten as a monotone transformation of the arithmetic mean, but that's probably because I don't know enough variational calculus (certainly not enough to prove I'm right). If I'm correct, however, then it would be "essentially" true that

in order for the theory of proper scoring rules to work as intended, the statistical functional must be the mean.

One other remark: you suggest using the RMSE as a scoring rule, but that you shouldn't do it because it coincides with the absolute error when there is one data point. This seems like it might reflect some confusion. You always evaluate a scoring rule on each individual prediction. Then if you want to summarize the scores, you can take the scores' central tendency afterwards. So predicting to optimize the RMSE is always identical to optimizing the absolute error.

On the other hand, you could do something like take the square root of the mean Brier score as your summary if you wanted to have a score summary that was in "units of probability." But I think it would be more productive to simply familiarize yourself with benchmarks for the Brier score scale, since that's what you'll typically see:

• 0 is a perfect predictor;
• 0.25 means no predictive ability ($\hat p = 0.5$);
• 1 is a perfect anti-predictor ($\hat p = 1, p = 0$ or $\hat p = 0, p = 1$).

You can also construct other benchmarks by using very simple models--for instance, if you ignore all info about the events and simply predict the base rate $p$, then your Brier score is $p(1-p)$. Or if you're predicting time series you can see how well a weighted average of the past few events does, etc.

• Thank you for your thoughtful reply. "then $g(\hat p) = \log f(\hat p)$ is a proper scoring rule for the geometric mean" — Do you mean $e^{f(\hat p)}$? Then we get $E[\log S_2] = E[\log e^S] = E[S]$, which has the same minimizing $\hat p$ as $e^{E(\log S_2)}$, as desired. – Kodiologist Apr 17 '15 at 13:18
• "You always evaluate a scoring rule on each individual prediction. Then if you want to summarize the scores, you can take the scores' central tendency afterwards." — In practice, there seem to be two phases in which a central tendency is involved: (1) when aggregating a single agent's scores across multiple events (2) when considering an agent's long-run performance. (2) uses the output of (1). One might've expected, a priori, that one could use RMSE for (1) but the mean for analyses regarding (2). – Kodiologist Apr 17 '15 at 13:27
• @Kodiologist: Thanks for the correction! Serves me right for not writing it out first. – Ben Kuhn Apr 17 '15 at 19:05
• Re your second comment: I think the confusion may be between using RMSE as a central tendency, and using RMSE as a scoring rule. As a scoring rule, the RMSE is identical to absolute error, because scores are evaluated on a prediction-by-prediction basis. As a central tendency, it's fine--it's again just the mean in a monotonically transformed coordinate space (as long as the scoring function is positive). But using RMSE as your central tendency (not scoring rule) does not alleviate the problem that your scores do not have the same units as probabilities. – Ben Kuhn Apr 17 '15 at 19:12
• @Kodiologist: does this answer your question? Let me know if you're still wondering anything! – Ben Kuhn Apr 21 '15 at 6:54

You have to go back to the motivation for a proper scoring rule, which you state loosely as "the agent with the least score makes the most accurate guesses." To be precise, the origin of scoring rules are to elicit probabilities that reflect true beliefs -as you state, a person can do no better than offer a probability corresponding to their belief when offered a scoring-rule as a reward. Scoring rules have been used to define what a probability means without referring to the limit of a large number of repetitions.

Such a scoring rule is derived by taking expectation over the rule, hence the appearance of the mean over the set of predictions. So when you ask must "the statistical functional must be the mean?" you are really asking how can we take the expectation over a set of scores by some other method than the conventional use of the mean?

I read into your concern that "proper scoring rules aren't on the same scale as probabilities" that perhaps you are looking to express how good or bad the computed score is? Aside from the Brier score, the log of the absolute difference between the offered probability and a 0,1 outcome is also a proper scoring rule, but that may not give more interpretable results, especially since it can diverge to extreme values for large errors.

Buried in the derivation of scoring rules is that the decision maker has linear utility, hence expectation is taken over the scoring rule directly, not over the utility of the scoring rule outcome. (A person may be risk adverse to large deviations from the truth, and that would bias their elicited probabilities.) Perhaps you are implicitly thinking of a utility function that expresses how good or bad are the "probabilities of what people will choose" instead of just the probabilities themselves?

• Re using scoring rules to define probability: interesting, I had no idea. Yes, looking to assess how good computed scores are is a concern for me. Re logarithmic scoring giving extreme scores for big errors: duly noted. Re nonlinear utility: you may be right, but deciding on utility functions seems like a very tricky business, especially in basic research. – Kodiologist Apr 21 '15 at 13:02