(Why) Is absolute loss not a proper scoring rule?

Question

Brier score is a proper scoring rule and is, at least in the binary classification case, square loss.

$$Brier(y,\hat{y}) = \frac{1}{N} \sum_{i=1}^N\big\vert y_i -\hat{y}_i\big\vert^2$$

Apparently this can be adjusted for when there are three or more classes.

In another post on Cross Validated, it is mentioned that absolute loss is not a proper scoring rule.

$$ absoluteLoss(y,\hat{y}) = \frac{1}{N} \sum_{i=1}^N\big\vert y_i -\hat{y}_i\big\vert $$

It seems similar enough to Brier score that it should be a proper scoring rule.

1. Why is absolute loss not a proper scoring rule?

2. Is absolute loss a proper scoring rule in the binary classification case that loses its "properness" when there are more than two output categories?

3. Can absolute loss be wrestled with like Brier score to have a proper form when there are more that two classes?

At least in the binary case, absolute loss has an easier interpretation than Brier score or the square root of Brier score in that it says the average amount by which a predicted probability differs from the observed outcome, so I would like to have a way for absolute loss to be proper.

Answer 1

Let's first make sure we agree on definitions. Consider a binary random variable $Y \sim \text{Ber}(p)$, and consider a loss function $L(y_i|s)$, where $s$ is an estimate of $p$ given the data. In your examples, $s$ is a function of observed data $y_1,\dots,y_n$ with $s = \hat{p}$. The Brier score loss function is $L_b(y_i | s) = |y_i - s|^2$, and the absolute loss function is $L_a(y_i|s) = |y_i - s|$. A loss function has an expected loss $E_Y(L(Y|s)) := R(p|s)$. A loss function is a proper score rule if the expected loss $R(p|s)$ is minimized with respect to $s$ by setting $s=p$ for any $p\in(0,1)$.

A handy trick for verifying this is using the binary nature of $Y$, as for any expected loss, we have $$R(p|s) = pL(1|s) + (1-p)L(0|s)$$

Let's start by verifying that the Bier loss function is a proper score rule. Note that $L_b(1|s) = |1-s|^2 = (1-s)^2$, and $L_b(0|s) = s^2$, so using the above, we have $$R_b(p|s) = p(1-s)^2 + (1-p)s^2$$

and taking derivative of that function wrt to $s$ and setting to $0$ will give you that the choice of $s = p$ minimizes the expected risk. So the Brier score is indeed a proper score rule.

In contrast, recalling the binary nature of $Y$, we can write the absolute loss $L_a$ as $$L_a(y|s) = y(1-s) + (1-y)s$$ as $y\in\{0,1\}$. As such, we have that $$R_a(p|s) = p(1-s) + (1-p)s = p + s - 2ps$$

Unfortunately, $R_a(p|s)$ is not minimized by $s=p$, and by considering edge cases, you can show that $R_a(p|s)$ is minimized by $s=1$ when $p>.5$, and by $s=0$ when $p<.5$, and holds for any choice of $s$ when $p=.5$.

So to answer your questions, absolute loss is not a proper scoring rule, and that does not have to the with the number of output categories. As for whether it can be wrestled, I certainly can't think of a way... I think such attempts to think of similar approaches will probably lead you to the Brier score :).

Edit:

In response to OP's comment, note that the absolute loss approach is basically estimating the median of $Y$, which in the binary case is in expectation either $0$ or $1$ depending on $p$. The absolute loss just doesn't penalize the alternative choice enough to make you want to choose anything but the value that shows up the most. In contrast, the squared error penalizes the alternative enough to find a middle ground that coincides with the mean $p$. This should also highlight that there's nothing wrong with using absolute loss as a classifier, and you can think of it related to determining, for a given problem, if you care more about the mean or the median. For binary data, I'd personally say the mean is more interesting (knowing the median tells you whether p > .5, but knowing the mean tells you a more precise statement about $p$), but it depends. As the other post also emphasizes, there's nothing wrong with absolute loss, it just isn't a proper score rule.

Answer 2

1. Take a simple example where $p_i$ are known probabilities and $y_i$ are Bernoulli($p_i$).

What is $\hat y_i$? The best choice is obviously $\hat y_i=p_i$. Alternatively, we might take $\check y_i = 1$ if $p_i>0.5$ and $\check y_i=0$ if $p_i<0.5$.

Suppose $p_i>0.5$ (for simplicity).

The expected Brier loss of $\hat y_i$ is $(1-p_i)^2p_i+p_i^2(1-p_i)=1-p_i^2$. The expected Brier loss of $\check y_i$ is $0^2\times p_i + 1^2\times (1-p_i)=1$, so $\hat y_i$ is preferred over $\check y_i$.

The expected absolute loss of $\hat y_i$ is $(1-p_i)p_i+p_i(1-p_i)=2p_i(1-p_i)$. The expected Brier loss of $\check y_i$ is $0\times p_i + 1\times (1-p_i)=1-p_i$, and since $p_i>0.5$, $2p_i(1-p_i)>(1-p_i)$ so $\check y_i$ is preferred over $\hat y_i$.

So, minimising absolute loss makes you say $\check y_i$ is better than true probability $\hat y_i$, which is what it means to be improper.

Note that $\check y_i$ is the median of $Y_i|p_i$, so it's not necessarily a bad estimator. And absolute error isn't necessarily a bad loss function. It's just not a proper scoring rule.

If you're going to have a continuous loss like this be proper it will have to penalise big errors more than small errors, so it will not have the interpretation you want it to have.

2. No, you get the same problems

3. No, you get the same problems

Answer 3

In a slightly different direction, one way to look at this is to consider more generally the continuous ranked probability score (CRPS), which is a proper scoring rule.

For a predicted CDF $F$ and an observation $y$, the CRPS is defined like this:

$$\text{CRPS}(F,y) = \int (F(z)-I(y\leq z))^2dz$$

Intuitively it is a measure of the distance between $F$ and a perfect predicted CDF which is exact and without uncertainty (i.e. $P[Y=y]=1$).

Let's restrict ourselves to $y$ being either 0 or 1. If our prediction $F$ is the CDF of a Bernoulli distribution with parameter $\hat{p}$, then you can show fairly easily that:

$$\text{CRPS}(F,y) = (y-\hat{p})^2$$

That is, the CRPS just reduces to the Brier score when the observations are 0-1 and $F$ is Bernoulli.

We'd like to find a distribution $F$ for which the CRPS reduces to absolute error instead. One possibility is to take the degenerate forecast $P[Y=\hat{y}]=1$. That is, this prediction is that $Y$ is not really random at all, and instead of being either 0 or 1, it is always $\hat{y}$. Then, we can show:

$$\text{CRPS}(F,y) = |y-\hat{y}|$$

As the other answers have shown, this is minimized at either $\hat{y}=0$ or $\hat{y}=1$. This shouldn't be particularly surprising; any other value means that, in our prediction $F$, the probability of observing either 0 or 1 is zero, which shouldn't give you a good score given that we've assumed those are the only possibilities.

Then, in the context of 0-1 data, minimizing the absolute error is kind of like minimizing CRPS (which is proper) but over a class of distributions which does not contain Bernoulli distributions with $0 < p < 1$, so isn't proper in general.