25
$\begingroup$

Merkle & Steyvers (2013) write:

To formally define a proper scoring rule, let $f$ be a probabilistic forecast of a Bernoulli trial $d$ with true success probability $p$. Proper scoring rules are metrics whose expected values are minimized if $f = p$.

I get that this is good because we want to encourage forecasters to generating forecasts that honestly reflect their true beliefs, and don't want to give them perverse incentives to do otherwise.

Are there any real-world examples in which it's appropriate to use an improper scoring rule?

Reference
Merkle, E. C., & Steyvers, M. (2013). Choosing a strictly proper scoring rule. Decision Analysis, 10(4), 292-304

$\endgroup$
  • 1
    $\begingroup$ I think the first column of the last page of Winkler & Jose "Scoring rules" (2010) which Merkle & Steyvers (2013) cite offers an answer. Namely, if utility is not an affine transformation of the score (which could be justified by risk aversion and such), maximization of expected utility would be in conflict with maximization of expected score $\endgroup$ – Richard Hardy Sep 19 at 12:18
24
$\begingroup$

It is appropriate to use an improper scoring rule when the purpose is actually forecasting, but not inference. I don't really care whether another forecaster is cheating or not when I am the one who is going to be doing the forecast.

Proper scoring rules ensure that during estimation process the model approaches the true data generating process (DGP). This sounds promising because as we approach the true DGP we will be also doing good in terms of forecasting under any loss function. The catch is that most of the time (actually in reality almost always) our model search space doesn't contain the true DGP. We end up approximating the true DGP with some functional form that we propose.

In this more realistic setting, if our forecasting task is easier than to figure out the entire density of the true DGP we may actually do better. This is especially true for classification. For example the true DGP can be very complex but the classification task can be very easy.

Yaroslav Bulatov provided the following example in his blog:

http://yaroslavvb.blogspot.ro/2007/06/log-loss-or-hinge-loss.html

As you can see below the true density is wiggly but it is very easy to build a classifier to separate data generated by this into two classes. Simply if $x \ge 0$ output class 1, and if $x < 0$ output class 2.

enter image description here

Instead of matching the exact density above we propose the below crude model, which is quite far from the true DGP. However it does perfect classification. This is found by using hinge loss, which is not proper.

enter image description here

On the other hand if you decide to find the true DGP with log-loss (which is proper) then you start fitting some functionals, as you don't know what the exact functional form you need a priori. But as you try harder and harder to match it, you start misclassifying things.

enter image description here

Note that in both cases we used the same functional forms. In the improper loss case it degenerated into a step function which in turn did perfect classification. In the proper case it went berserk trying to satisfy every region of the density.

Basically we don't always need to achieve the true model to have accurate forecasts. Or sometimes we don't really need to do good on the entire domain of the density, but be very good only on certain parts of it.

$\endgroup$
  • 13
    $\begingroup$ That is a fascinating example, really some food for thought. $\endgroup$ – Matthew Drury Dec 14 '16 at 21:59
7
$\begingroup$

Accuracy (i.e., percent correctly classified) is an improper scoring rule, so in some sense people do it all the time.

More generally, any scoring rule that forces predictions into a pre-defined category is going to be improper. Classification is an extreme case of this (the only allowable forecasts are 0% and 100%), but the weather forecast is probably also slightly improper--my local stations seems to report the chance of rain in 10 or 20% intervals, though I'd bet the underlying model is much more precise.

Proper scoring rules also assume that the forecaster is risk neutral. This is often not the case for actual human forecasters, who are typically risk-adverse, and some applications might benefit from a scoring rule that reproduces that bias. For example, you might give a little extra weight to P(rain) since carrying an umbrella but not needing it is far better than being caught in a downpour.

$\endgroup$
  • 3
    $\begingroup$ I don't think I understand your third paragraph. I had been writing up a similar answer along the lines that we may want to concentrate more on getting high quantiles of predictive densities right, but I don't see how such a loss function would motivate us to use an improper scoring rule. We'd still be most motivated to forecast the correct future distribution, after all. Could you elaborate? $\endgroup$ – Stephan Kolassa Apr 21 '16 at 8:19
  • 1
    $\begingroup$ If the forecaster maximizes its expected utility (instead of value), then the proper scoring rules may not actually be proper (e.g., if the utility isn't a linear function of the score). However if you know or can estimate the utility function, I guess you could come up with a specially-tailored proper scoring rule instead by applying its inverse. $\endgroup$ – Matt Krause Apr 21 '16 at 9:04
  • 3
    $\begingroup$ But the properness or not of the scoring rule is not related to the utility, only to the predicted and actual future distribution, so I'm still not understanding the first sentence of your comment, nor why we would want to use an improper scoring rule. However, you remind me of a paper by Ehm at al, to appear in JRSS-B, which I skimmed in writing my aborted answer, but where I didn't find anything useful for the present question - closer reading may be more helpful. $\endgroup$ – Stephan Kolassa Apr 21 '16 at 9:57
  • $\begingroup$ @StephanKolassa, perhaps the first column of the last page of Winkler & Jose "Scoring rules" (2010) explains it? $\endgroup$ – Richard Hardy Sep 19 at 11:51
  • $\begingroup$ Comments are not for extended discussion; this conversation has been moved to chat. $\endgroup$ – gung Sep 20 at 11:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.