When is it appropriate to use an improper scoring rule?

Question

Merkle & Steyvers (2013) write:

To formally define a proper scoring rule, let $f$ be a probabilistic forecast of a Bernoulli trial $d$ with true success probability $p$. Proper scoring rules are metrics whose expected values are minimized if $f = p$.

I get that this is good because we want to encourage forecasters to generating forecasts that honestly reflect their true beliefs, and don't want to give them perverse incentives to do otherwise.

Are there any real-world examples in which it's appropriate to use an improper scoring rule?

Reference
Merkle, E. C., & Steyvers, M. (2013). Choosing a strictly proper scoring rule. Decision Analysis, 10(4), 292-304

Answer 1

It is appropriate to use an improper scoring rule when the purpose is actually forecasting, but not inference. I don't really care whether another forecaster is cheating or not when I am the one who is going to be doing the forecast.

Proper scoring rules ensure that during estimation process the model approaches the true data generating process (DGP). This sounds promising because as we approach the true DGP we will be also doing good in terms of forecasting under any loss function. The catch is that most of the time (actually in reality almost always) our model search space doesn't contain the true DGP. We end up approximating the true DGP with some functional form that we propose.

In this more realistic setting, if our forecasting task is easier than to figure out the entire density of the true DGP we may actually do better. This is especially true for classification. For example the true DGP can be very complex but the classification task can be very easy.

Yaroslav Bulatov provided the following example in his blog:

http://yaroslavvb.blogspot.ro/2007/06/log-loss-or-hinge-loss.html

As you can see below the true density is wiggly but it is very easy to build a classifier to separate data generated by this into two classes. Simply if $x \ge 0$ output class 1, and if $x < 0$ output class 2.

Instead of matching the exact density above we propose the below crude model, which is quite far from the true DGP. However it does perfect classification. This is found by using hinge loss, which is not proper.

On the other hand if you decide to find the true DGP with log-loss (which is proper) then you start fitting some functionals, as you don't know what the exact functional form you need a priori. But as you try harder and harder to match it, you start misclassifying things.

Note that in both cases we used the same functional forms. In the improper loss case it degenerated into a step function which in turn did perfect classification. In the proper case it went berserk trying to satisfy every region of the density.

Basically we don't always need to achieve the true model to have accurate forecasts. Or sometimes we don't really need to do good on the entire domain of the density, but be very good only on certain parts of it.

Answer 2

Accuracy (i.e., percent correctly classified) is an improper scoring rule, so in some sense people do it all the time.

More generally, any scoring rule that forces predictions into a pre-defined category is going to be improper. Classification is an extreme case of this (the only allowable forecasts are 0% and 100%), but the weather forecast is probably also slightly improper--my local stations seems to report the chance of rain in 10 or 20% intervals, though I'd bet the underlying model is much more precise.

Proper scoring rules also assume that the forecaster is risk neutral. This is often not the case for actual human forecasters, who are typically risk-adverse, and some applications might benefit from a scoring rule that reproduces that bias. For example, you might give a little extra weight to P(rain) since carrying an umbrella but not needing it is far better than being caught in a downpour.

Answer 3

A simplified answer, as indicated by Cagdas Ozgenc, might be: whenever you do not aim for the true predictive distribution.

A second aspect is the difference between fitting/estimation, inference, and forecast comparison. When you fit by minimizing a proper scoring rule and then add a penalty to deal with overfitting, your objective is usually no longer a proper scoring rule.

Thirdly, I'm not aware of use cases where you want a predictive distribution but not the true one, or as close as possible. Often in practice, however, you are content with predicting a certain functional of the predictive distribution, i.e. a point forecast like the expected value or a quantile. In those cases, the usage of proper scoring functions is advisable, unless there is a clear (business) objective that you want to optimize directly. Also note that the notions of scoring rule and scoring function for the expectation coincide for binary targets.

This is the general direction of Cagdas Ozgenc's answer, on which I'd like to comment (as I can't comment directly...yet):

1. The same parametric model should have been used for log-loss and hinge loss. I'm sure, a logistic regression with intercept plus feature $I_{x>0}$ would not be worse than the hinge example.
2. "However it does perfect classification." It does not. BTW, by which notion of "good classification"?
3. I guess by "classification", a concrete decision based on a (probabilistic) forecast is meant, have a look at https://stats.stackexchange.com/q/312787 for more details on that distinction.