What exactly does a proper scoring rule want to do?

Question

I will adapt an excellent simulation by our Stéphane Laurent for this question.

x1 <- c(0,0,1,1)                  # binary predictor #1
x2 <- c(0,1,0,1)                  # binary predictor #2
z <- 1 + 2*x1 - 3*x2              # linear combination with a bias
pr <- 1/(1+exp(-z))               # pass through an inv-logit function
y <- rbinom(length(pr),1,pr)      # Bernoulli response variable
round(pr,2)

The setup is that I have two binary predictor variables and a binary response variable, and I want to fit a model of the response variable, probably logistic regression.

I assess my model with a proper scoring rule. What does the proper scoring rule want to achieve, perfect accuracy (all $0$s called $P(1)=0$ and all $1$s called $P(1)=1$) or the perfect probability at the four combinations of predictors?

Perfect probability of predictors:

$$P(Y=1\vert x_1=0, x_2=0) = 0.73$$ $$P(Y=1\vert x_1=0, x_2=1) = 0.12$$ $$P(Y=1\vert x_1=1, x_2=0) = 0.95$$ $$P(Y=1\vert x_1=1, x_2=1) = 0.50$$

This idea can be extended to models with continuous predictors, but two binary predictors makes it easy to give all of the possible combinations of predictors.

(Typing out this question, I think it has to be the latter case, the true probabilities, but it sure would be nice to get confirmation.)

EDIT

After discussing proper scoring rules on the data science Stack, I now have doubts about my parenthetical comment at the end of the original post. How does a proper scoring rule both want to find the true probabilities and optimize according to observed classes?

EDIT 2

The Brier score, for instance, is minimized when the categories are correctly predicted as $0$ and $1$. How is that related to finding the "true" probabilities?

In my simulation, if I predict $P(Y=1\vert x_1=0, x_2=0) = 0.73$, I get penalized by the Brier score, since I would have the true category be either $0$ or $1$. Or is the idea that, if I sampled many times from $x_1=0, x_2=0$ that I would get $73\%$ of the observations to be $1$ and $27\%$ of the observations to be $0$, so the best prediction is $0.73$ instead of a pure $0$ or $1$?

(Now I think I see what's going on, but it would be great to have someone confirm!)

Answer 1

Your thinking is correct. I recommend Gneiting & Raftery (2007, JASA) for an in-depth discussion of scoring rules.

A scoring rule $S$ is a mapping that takes a probabilistic prediction $\hat{p}$ and a corresponding observed outcome $y$ to a loss value $S(\hat{p},y)$. In our application, $\hat{p}$ is just a single number (that will depend on predictors, see below), but in a numerical prediction, it will be an entire predictive density. We typically take averages of this loss value over multiple instances $y_i$, each with its own (predictor-dependent) prediction $\hat{p}_i$. And we usually aim at minimizing this average loss (though the opposite convention also exists; it's always a good idea to verify how a particular paper's scoring rules are oriented).

A scoring rule is proper if it is minimized in expectation by the true probability.

Now, in the present case, the key aspect is that we have only two predictors, both of which can only take the values $0$ and $1$. In this setting, we cannot distinguish between two instances with different outcomes $y$ but the same predictor settings, so we cannot have different (probabilistic) predictions for two instances with the same predictor settings. Having a hard $0$ prediction for an instance with $y=0$, but a hard $1$ prediction for an instance with $y=1$ is simply not possible if the two instances have the same predictor values. All we can have is a probabilistic prediction $\hat{p}_{ij}$ in the case where the first predictor has value $i$ and the second predictor has value $j$.

Now, let's assume that the true probability of $y=1$, given that the first predictor has value $i$ and the second predictor has value $j$, is $p_{ij}$. What is the expected value of the Brier score of our probabilistic prediction $\hat{p}_{ij}$?

Well, with a probability of $p_{ij}$, we have $y=1$ and a contribution of $(1-\hat{p}_{ij})^2$ to the Brier score, and with a probability of $1-p_{ij}$, we have $y=0$ and a contribution of $\hat{p}_{ij}^2$ to the Brier score. The total expected constribution to the Brier score is

$$ p_{ij}(1-\hat{p}_{ij})^2+(1-p_{ij})\hat{p}_{ij}^2. $$

Differentiating this expression with respect to $\hat{p}_{ij}$ and setting the derivative equal to zero, we find that this expected score is minimized when $\hat{p}_{ij}=p_{ij}$, so we have found that the Brier score is proper in our situation. It aims at getting the correct (specifically: calibrated and sharp) probabilistic prediction.

And of course, if now a third predictor turns up that would allow perfect $0-1$ predictions, then the Brier score of this expanded model would be lower than that of the two-predictor model's predictions (namely, zero). Which is exactly how it should be.

Answer 2

I kind of figured out the answer as I was writing the question, so a few days later, I'm going to write a self-answer.

The right answer is that the distribution of $Y$ when $x_1=0$ and $x_2=0$ is that $P(Y=1) = 0.73$. We get a discrete outcome of either a $0$ or a $1$, but the model should be able figure out that the distribution is $Bernoulli(0.73)$. If the model misses this fact, the (proper) scoring rule should penalize the model.

Ditto for the other combinations of $x_1$ and $x_2$.

Thinking about a more real-life problem, a photograph might look like it is of a dog, and perhaps there is a $99.99\%$ chance of being a dog; given $10,000$ such photos, all but $1$ would be a dog. However, one of those times, it would happen to be a cat. A good model will pick up on the fact that such a photo could, occasionally, be of a cat, and the proper scoring wants to find the model that is able to say that a cat might occasionally look like that dog.

Answer 3

Scoring rules assess the quality of a probabilistic forecast; i.e. a prediction with some uncertainty measure associated to it. This could be something simple like a mean and standard deviation, or it could be a full probability distribution (or something in between!). The idea behind a (proper) scoring rule is to encourage 'honest' probabilistic predictions. Suppose I am estimating an unknown parameter $\theta$ by some probability distribution $P(\hat{\theta})$, and suppose we are using a positively oriented score (bigger is better). I will increase my score if

• The mean implied by $P(\hat{\theta})$ is close to $\theta$ and the uncertainty is relatively small
• The mean implied by $P(\hat{\theta})$ is far from $\theta$ but my uncertainty is relatively large

If I get small uncertainty with large error, I will have poor score. Likewise, an accurate but uncertain forecast will be penalised.

Essentially, I am trying to create a well-calibrated forecast. I am embracing uncertainty, and trying to identify an appropriate amount of uncertainty in my predictions.