Minimizing expected brier score and Brier score interpretation

Question

For a probabilistic binary forecast, the BS (Brier score) is given by $$ \text{BS}= \begin{cases} (1-f_i)^2\\ f_i^2\\ \end{cases} $$ Where $f$ is the forecast. If the event occurs with probability $p_i$ then the Expected Brier score is $$E[\text{BS}] = p_i(1-f_i)^2 + (1-p_i)f_i^2$$ which is minimized by setting $f = p$. This means that if one where to make accurate forecast $f$ of the true probability the expected Brier score reaches a minimum.

If we instead had many probabilistic forecasts, $\text{BS}=\sum \text{BS}_i$, then its expectation would be minimized when every forecast equals the true probability for the outcome.

If the random variable $\text{BS}$ materializes the sample mean is:$n^{-1} \sum (f_i-O_i)^2$. Where $O_i$ is the observed event: 1 or 0.

1. But the sample mean is minimized by letting $f_i$ equal the true outcome: 1 or 0 which may not be the true probability of the outcome. Something is wrong with my reasoning but I can't understand what? Could someone explain?

2. From the reasoning about minimizing the expected Brier score above, should I interpret the Brier score such that if I minimized the expected Brier score then I am making more accurate predictions?

** EDITED** I Want to emphasize that each event has a different probability of occurring.

** EDITED** @kjetil b halvorsen

suppose we fitted a logistic regression in millions of observations then we fit the model $logit( f_i) = \hat{\alpha} + \hat{\beta}_1 x $

What is the difference when we use logistic regression model? what more restrictions are there than less parameters than observations?

In this setting we probably could not minimize the sample mean so that it equals zero ?

Answer 1

You have cast the problem in terms of a series of identically distributed binary observations, i.e., every observation has the same probability that $Y=1$. The mean of the binary values, i.e, the overall proportion of $Y=1$ minimizes the Brier score. This is also the maximum likelihood estimator for the Bernoulli (binary $Y$) distribution. From the data you don't know the true probability of the outcome but in the absence of external data the overall proportion is the best estimate of it.

In general we use the Brier score to judge the quality of probability estimates, but use the likelihood function as the objective function for estimation. Generalizing to non-equal probabilities across units we use models such as the binary logistic model in this heterogeneous setting.

Answer 2

What is wrong with your reasoning? In 1. you minimize the expectation of each term before taking the sum. You say

I Want to emphasize that each event has a different probability of occurring

and in that case, well, it is correct to minimize each term individually, but you have one parameter for each observation. Logistic regression means to model those $p_i$'s by letting them depend on some covariates, you haven't told us about covariates. Lacking them, you can do no better than the rather unhelpful estimates of 0 or 1.

Question 2. is potentially more interesting. You ask

should I interpret the Brier score such that if I minimized the expected Brier score then I am making more accurate predictions?

Well, the Brier score is just one of many other proper scoring rules and then the question becomes which proper scoring rule gives "best" predictions? Then of course you would need to define what you mean by "more accurate predictions". One paper asking this question is Choosing a Strictly Proper Scoring Rule. One could also ask if there is a theory of using proper scoring rules as inference functions, one paper is Minimum Scoring Rule Inference.