Probabilistic classification and loss functions

Question

I am trying to compare several binary classifiers. These classifiers (Gaussian Processes in my case, but it shouldn't matter) give me probabilistic predictions. Let's introduce some notations:

$$y_i \in \{0,1\} = \text{true class of sample $i$}$$ $$p_i \in [0,1] = \text{Pr}(y_i = 1 \mid x_i), \text{probability that sample $i$ is positive}$$ I know two "classical" ways to measure the accuracy of the probabilistic classifiers:

1. 0-1 loss: just predict the class that has highest probability, and compare against the true class: $$L_i = y_i\mathbf{1}_{\{p_i \le 0.5\}} + (1 - y_i)\mathbf{1}_{\{p_i > 0.5\}}$$
2. logarithmic loss: takes into account the predictive probability of the class. $$L_i = -y_i\log(p_i) - (1 -y_i) \log(1 -p_i)$$ This makes sense to me in that it is the log-likelihood of the sample, under the distribution induced by the classifier.

Now I am suggesting a third loss function, let's call it the "probability loss": $$L_i = y_i(1 - p_i) + (1 - y_i) p_i$$

I have a bunch of questions related to this last loss function:

1. Did I just make it up, or is it a well-known loss function?
2. Intuitively, i would interpret it as the "expected" 0-1 loss. Does this make sense?

And most importantly: where's the catch? I'm quite confident that this loss function is flawed, but I can't see where & how.

Note: this question seems related, in particular the comment to the first answer.

Answer 1

The "probability loss" function has sometimes been called the "linear score" in the literature. Although it looks appealing, this loss function is improper, which means that it does not set the incentive to forecast the true probability that $y_i = 1$. For details, see p. 366 of Gneiting and Raftery ("Strictly Proper Scoring Rules, Prediction, and Estimation", Journal of the American Statistical Association, 2007).

In practice, impropriety means that a silly forecaster (who, for example, skews his probabilities towards the extremes of zero and one) may obtain a better probability loss than a reasonable forecaster.

The following example, based on R code, illustrates this point.

• First, set a random seed (set.seed(1)) and fix a sample size (n <- 10000)
• Simulate an arbitrary vector of true probabilities: p_true <- runif(n)
• Now, draw a vector of binary observations which follow these probabilities: y <- runif(n) < p_true
• Suppose Anne is an omniscient forecaster, and knows the true probabilities p_true. Her probability loss (on average over cases) can be computed as follows: loss_Anne <- (sum(p_true[y == FALSE]) + sum((1-p_true)[y == TRUE]))/n
• By contrast, consider a second forecaster (Bob) who makes overconfident predictions, according to the following formula: p_wrong <- 0.5*(p_true + (p_true >= 0.5)). The formula means that Bob skews the "small" probabilities (less than 50 percent) towards zero, and the "large" probabilities (more than 50 percent) towards one. Bob's average loss is given by loss_Bob <- (sum(p_wrong[y == FALSE]) + sum((1-p_wrong)[y == TRUE]))/n

Running this code on my PC, I find that loss_Anne is about $0.33$, whereas loss_Bob is about $0.29$. Thus, a perfect forecaster (Anne) loses to an overconfident forecaster who deliberately skews his probabilities towards the extremes of zero and one.

Thus, the probability loss should not be used for model comparison, as it will generally not select the true model (even asymptotically). Instead, a strictly proper scoring function like the logarithmic loss or Brier score should be used. Again see the reference mentioned above.