# When is it appropriate to use an improper scoring rule?

Merkle & Steyvers (2013) write:

To formally define a proper scoring rule, let $f$ be a probabilistic forecast of a Bernoulli trial $d$ with true success probability $p$. Proper scoring rules are metrics whose expected values are minimized if $f = p$.

I get that this is good because we want to encourage forecasters to generating forecasts that honestly reflect their true beliefs, and don't want to give them perverse incentives to do otherwise.

Are there any real-world examples in which it's appropriate to use an improper scoring rule?

• I think the first column of the last page of Winkler & Jose "Scoring rules" (2010) which Merkle & Steyvers (2013) cite offers an answer. Namely, if utility is not an affine transformation of the score (which could be justified by risk aversion and such), maximization of expected utility would be in conflict with maximization of expected score Sep 19, 2019 at 12:18
• The idea of wet bias is related. Some weather forecasters deliberately overstate the probability of rain because people get more upset when the forecast says it probably won't rain but it actually does rain than when the reverse happens. Jun 6, 2021 at 21:56

It is appropriate to use an improper scoring rule when the purpose is actually forecasting, but not inference. I don't really care whether another forecaster is cheating or not when I am the one who is going to be doing the forecast.

Proper scoring rules ensure that during estimation process the model approaches the true data generating process (DGP). This sounds promising because as we approach the true DGP we will be also doing good in terms of forecasting under any loss function. The catch is that most of the time (actually in reality almost always) our model search space doesn't contain the true DGP. We end up approximating the true DGP with some functional form that we propose.

In this more realistic setting, if our forecasting task is easier than to figure out the entire density of the true DGP we may actually do better. This is especially true for classification. For example the true DGP can be very complex but the classification task can be very easy.

Yaroslav Bulatov provided the following example in his blog:

http://yaroslavvb.blogspot.ro/2007/06/log-loss-or-hinge-loss.html

As you can see below the true density is wiggly but it is very easy to build a classifier to separate data generated by this into two classes. Simply if $$x \ge 0$$ output class 1, and if $$x < 0$$ output class 2.

Instead of matching the exact density above we propose the below crude model, which is quite far from the true DGP. However it does perfect classification. This is found by using hinge loss, which is not proper.

On the other hand if you decide to find the true DGP with log-loss (which is proper) then you start fitting some functionals, as you don't know what the exact functional form you need a priori. But as you try harder and harder to match it, you start misclassifying things.

Note that in both cases we used the same functional forms. In the improper loss case it degenerated into a step function which in turn did perfect classification. In the proper case it went berserk trying to satisfy every region of the density.

Basically we don't always need to achieve the true model to have accurate forecasts. Or sometimes we don't really need to do good on the entire domain of the density, but be very good only on certain parts of it.

• That is a fascinating example, really some food for thought. Dec 14, 2016 at 21:59
• @CagdasOzgenc I have tried implementing this example and unfortunately I don't think it is correct. I think the blog author has used the hinge loss to fit a logistic regression model to the probability of class membership, rather than to data drawn from the implied distribution. That way all of the data to the left of x=0 have p < 0.5 and all of the data above have p > 0.5. However if you sample data from that distribution you will have labels of 0 and 1 on both sides of x=0 and you do not get the result shown. So unfortunately without that detail it is a bit misleading. Mar 22, 2022 at 14:36
• Have done a bit more work on this and posted it here: stats.stackexchange.com/questions/568821/… It still works as a demo that an improper scoring rule can give better accuracy, but it isn't nearly as clear cut as the example from the blog. Mar 29, 2022 at 16:01

Accuracy (i.e., percent correctly classified) is an improper scoring rule, so in some sense people do it all the time.

More generally, any scoring rule that forces predictions into a pre-defined category is going to be improper. Classification is an extreme case of this (the only allowable forecasts are 0% and 100%), but the weather forecast is probably also slightly improper--my local stations seems to report the chance of rain in 10 or 20% intervals, though I'd bet the underlying model is much more precise.

Proper scoring rules also assume that the forecaster is risk neutral. This is often not the case for actual human forecasters, who are typically risk-adverse, and some applications might benefit from a scoring rule that reproduces that bias. For example, you might give a little extra weight to P(rain) since carrying an umbrella but not needing it is far better than being caught in a downpour.

• I don't think I understand your third paragraph. I had been writing up a similar answer along the lines that we may want to concentrate more on getting high quantiles of predictive densities right, but I don't see how such a loss function would motivate us to use an improper scoring rule. We'd still be most motivated to forecast the correct future distribution, after all. Could you elaborate? Apr 21, 2016 at 8:19
• If the forecaster maximizes its expected utility (instead of value), then the proper scoring rules may not actually be proper (e.g., if the utility isn't a linear function of the score). However if you know or can estimate the utility function, I guess you could come up with a specially-tailored proper scoring rule instead by applying its inverse. Apr 21, 2016 at 9:04
• But the properness or not of the scoring rule is not related to the utility, only to the predicted and actual future distribution, so I'm still not understanding the first sentence of your comment, nor why we would want to use an improper scoring rule. However, you remind me of a paper by Ehm at al, to appear in JRSS-B, which I skimmed in writing my aborted answer, but where I didn't find anything useful for the present question - closer reading may be more helpful. Apr 21, 2016 at 9:57
• @StephanKolassa, perhaps the first column of the last page of Winkler & Jose "Scoring rules" (2010) explains it? Sep 19, 2019 at 11:51
• Comments are not for extended discussion; this conversation has been moved to chat. Sep 20, 2019 at 11:28

A simplified answer, as indicated by Cagdas Ozgenc, might be: whenever you do not aim for the true predictive distribution.

A second aspect is the difference between fitting/estimation, inference, and forecast comparison. When you fit by minimizing a proper scoring rule and then add a penalty to deal with overfitting, your objective is usually no longer a proper scoring rule.

Thirdly, I'm not aware of use cases where you want a predictive distribution but not the true one, or as close as possible. Often in practice, however, you are content with predicting a certain functional of the predictive distribution, i.e. a point forecast like the expected value or a quantile. In those cases, the usage of proper scoring functions is advisable, unless there is a clear (business) objective that you want to optimize directly. Also note that the notions of scoring rule and scoring function for the expectation coincide for binary targets.

This is the general direction of Cagdas Ozgenc's answer, on which I'd like to comment (as I can't comment directly...yet):

1. The same parametric model should have been used for log-loss and hinge loss. I'm sure, a logistic regression with intercept plus feature $$I_{x>0}$$ would not be worse than the hinge example.
2. "However it does perfect classification." It does not. BTW, by which notion of "good classification"?
3. I guess by "classification", a concrete decision based on a (probabilistic) forecast is meant, have a look at https://stats.stackexchange.com/q/312787 for more details on that distinction.
• Well, it does do perfect classification as should be clear from the graph. It always guesses the class correctly. This is good by any sensible measure of quality of classification. Jun 6, 2021 at 16:36
• This is really a great example. If I read the y-axis correctly, it is $p(y=1|x)$, i.e. it is the conditional probability. Let's focus on the point $x\approx0.25$ with $p(y=1|x=0.25)=0.5$. The hinge model does not classify perfectly as it is wrong every second time (although it's hard to to better). But it's even worse because it is not calibrated. This leads again to the distinction between probabilistic forecast and decision making well discussed in stats.stackexchange.com/q/312787. Jun 6, 2021 at 19:30