Merkle & Steyvers (2013) write:

To formally define a proper scoring rule, let $f$ be a probabilistic forecast of a Bernoulli trial $d$ with true success probability $p$. Proper scoring rules are metrics whose expected values are minimized if $f = p$.

I get that this is good because we want to encourage forecasters to generating forecasts that honestly reflect their true beliefs, and don't want to give them perverse incentives to do otherwise.

Are there any real-world examples in which it's appropriate to use an improper scoring rule?

Merkle, E. C., & Steyvers, M. (2013). Choosing a strictly proper scoring rule. Decision Analysis, 10(4), 292-304

  • 1
    $\begingroup$ I think the first column of the last page of Winkler & Jose "Scoring rules" (2010) which Merkle & Steyvers (2013) cite offers an answer. Namely, if utility is not an affine transformation of the score (which could be justified by risk aversion and such), maximization of expected utility would be in conflict with maximization of expected score $\endgroup$ Sep 19 '19 at 12:18
  • 1
    $\begingroup$ The idea of wet bias is related. Some weather forecasters deliberately overstate the probability of rain because people get more upset when the forecast says it probably won't rain but it actually does rain than when the reverse happens. $\endgroup$
    – fblundun
    Jun 6 at 21:56

It is appropriate to use an improper scoring rule when the purpose is actually forecasting, but not inference. I don't really care whether another forecaster is cheating or not when I am the one who is going to be doing the forecast.

Proper scoring rules ensure that during estimation process the model approaches the true data generating process (DGP). This sounds promising because as we approach the true DGP we will be also doing good in terms of forecasting under any loss function. The catch is that most of the time (actually in reality almost always) our model search space doesn't contain the true DGP. We end up approximating the true DGP with some functional form that we propose.

In this more realistic setting, if our forecasting task is easier than to figure out the entire density of the true DGP we may actually do better. This is especially true for classification. For example the true DGP can be very complex but the classification task can be very easy.

Yaroslav Bulatov provided the following example in his blog:


As you can see below the true density is wiggly but it is very easy to build a classifier to separate data generated by this into two classes. Simply if $x \ge 0$ output class 1, and if $x < 0$ output class 2.

enter image description here

Instead of matching the exact density above we propose the below crude model, which is quite far from the true DGP. However it does perfect classification. This is found by using hinge loss, which is not proper.

enter image description here

On the other hand if you decide to find the true DGP with log-loss (which is proper) then you start fitting some functionals, as you don't know what the exact functional form you need a priori. But as you try harder and harder to match it, you start misclassifying things.

enter image description here

Note that in both cases we used the same functional forms. In the improper loss case it degenerated into a step function which in turn did perfect classification. In the proper case it went berserk trying to satisfy every region of the density.

Basically we don't always need to achieve the true model to have accurate forecasts. Or sometimes we don't really need to do good on the entire domain of the density, but be very good only on certain parts of it.

  • 16
    $\begingroup$ That is a fascinating example, really some food for thought. $\endgroup$ Dec 14 '16 at 21:59

Accuracy (i.e., percent correctly classified) is an improper scoring rule, so in some sense people do it all the time.

More generally, any scoring rule that forces predictions into a pre-defined category is going to be improper. Classification is an extreme case of this (the only allowable forecasts are 0% and 100%), but the weather forecast is probably also slightly improper--my local stations seems to report the chance of rain in 10 or 20% intervals, though I'd bet the underlying model is much more precise.

Proper scoring rules also assume that the forecaster is risk neutral. This is often not the case for actual human forecasters, who are typically risk-adverse, and some applications might benefit from a scoring rule that reproduces that bias. For example, you might give a little extra weight to P(rain) since carrying an umbrella but not needing it is far better than being caught in a downpour.

  • 3
    $\begingroup$ I don't think I understand your third paragraph. I had been writing up a similar answer along the lines that we may want to concentrate more on getting high quantiles of predictive densities right, but I don't see how such a loss function would motivate us to use an improper scoring rule. We'd still be most motivated to forecast the correct future distribution, after all. Could you elaborate? $\endgroup$ Apr 21 '16 at 8:19
  • 1
    $\begingroup$ If the forecaster maximizes its expected utility (instead of value), then the proper scoring rules may not actually be proper (e.g., if the utility isn't a linear function of the score). However if you know or can estimate the utility function, I guess you could come up with a specially-tailored proper scoring rule instead by applying its inverse. $\endgroup$ Apr 21 '16 at 9:04
  • 3
    $\begingroup$ But the properness or not of the scoring rule is not related to the utility, only to the predicted and actual future distribution, so I'm still not understanding the first sentence of your comment, nor why we would want to use an improper scoring rule. However, you remind me of a paper by Ehm at al, to appear in JRSS-B, which I skimmed in writing my aborted answer, but where I didn't find anything useful for the present question - closer reading may be more helpful. $\endgroup$ Apr 21 '16 at 9:57
  • $\begingroup$ @StephanKolassa, perhaps the first column of the last page of Winkler & Jose "Scoring rules" (2010) explains it? $\endgroup$ Sep 19 '19 at 11:51
  • $\begingroup$ Comments are not for extended discussion; this conversation has been moved to chat. $\endgroup$ Sep 20 '19 at 11:28

A simplified answer, as indicated by Cagdas Ozgenc, might be: whenever you do not aim for the true predictive distribution.

A second aspect is the difference between fitting/estimation, inference, and forecast comparison. When you fit by minimizing a proper scoring rule and then add a penalty to deal with overfitting, your objective is usually no longer a proper scoring rule.

Thirdly, I'm not aware of use cases where you want a predictive distribution but not the true one, or as close as possible. Often in practice, however, you are content with predicting a certain functional of the predictive distribution, i.e. a point forecast like the expected value or a quantile. In those cases, the usage of proper scoring functions is advisable, unless there is a clear (business) objective that you want to optimize directly. Also note that the notions of scoring rule and scoring function for the expectation coincide for binary targets.

This is the general direction of Cagdas Ozgenc's answer, on which I'd like to comment (as I can't comment directly...yet):

  1. The same parametric model should have been used for log-loss and hinge loss. I'm sure, a logistic regression with intercept plus feature $I_{x>0}$ would not be worse than the hinge example.
  2. "However it does perfect classification." It does not. BTW, by which notion of "good classification"?
  3. I guess by "classification", a concrete decision based on a (probabilistic) forecast is meant, have a look at https://stats.stackexchange.com/q/312787 for more details on that distinction.
  • $\begingroup$ Well, it does do perfect classification as should be clear from the graph. It always guesses the class correctly. This is good by any sensible measure of quality of classification. $\endgroup$ Jun 6 at 16:36
  • 1
    $\begingroup$ This is really a great example. If I read the y-axis correctly, it is $p(y=1|x)$, i.e. it is the conditional probability. Let's focus on the point $x\approx0.25$ with $p(y=1|x=0.25)=0.5$. The hinge model does not classify perfectly as it is wrong every second time (although it's hard to to better). But it's even worse because it is not calibrated. This leads again to the distinction between probabilistic forecast and decision making well discussed in stats.stackexchange.com/q/312787. $\endgroup$ Jun 6 at 19:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.