3
$\begingroup$

For a probabilistic binary forecast, the BS (Brier score) is given by $$ \text{BS}= \begin{cases} (1-f_i)^2\\ f_i^2\\ \end{cases} $$ Where $f$ is the forecast. If the event occurs with probability $p_i$ then the Expected Brier score is $$E[\text{BS}] = p_i(1-f_i)^2 + (1-p_i)f_i^2$$ which is minimized by setting $f = p$. This means that if one where to make accurate forecast $f$ of the true probability the expected Brier score reaches a minimum.

If we instead had many probabilistic forecasts, $\text{BS}=\sum \text{BS}_i$, then its expectation would be minimized when every forecast equals the true probability for the outcome.

If the random variable $\text{BS}$ materializes the sample mean is:$n^{-1} \sum (f_i-O_i)^2$. Where $O_i$ is the observed event: 1 or 0.

  1. But the sample mean is minimized by letting $f_i$ equal the true outcome: 1 or 0 which may not be the true probability of the outcome. Something is wrong with my reasoning but I can't understand what? Could someone explain?

  2. From the reasoning about minimizing the expected Brier score above, should I interpret the Brier score such that if I minimized the expected Brier score then I am making more accurate predictions?

** EDITED** I Want to emphasize that each event has a different probability of occurring.

** EDITED** @kjetil b halvorsen

suppose we fitted a logistic regression in millions of observations then we fit the model $logit( f_i) = \hat{\alpha} + \hat{\beta}_1 x $

What is the difference when we use logistic regression model? what more restrictions are there than less parameters than observations?

In this setting we probably could not minimize the sample mean so that it equals zero ?

$\endgroup$
  • 1
    $\begingroup$ To the second edit: The difference is that the expected value with respect to which the forecast minimises the brier score conditions on information you can actually use to make a forecast. If that information included the actual outcome, the $\hat{p_i}$ would be 0 or 1, but when we forecast an outcome, we don't know what it is $\endgroup$ – CloseToC Jul 11 at 12:11
3
$\begingroup$

You have cast the problem in terms of a series of identically distributed binary observations, i.e., every observation has the same probability that $Y=1$. The mean of the binary values, i.e, the overall proportion of $Y=1$ minimizes the Brier score. This is also the maximum likelihood estimator for the Bernoulli (binary $Y$) distribution. From the data you don't know the true probability of the outcome but in the absence of external data the overall proportion is the best estimate of it.

In general we use the Brier score to judge the quality of probability estimates, but use the likelihood function as the objective function for estimation. Generalizing to non-equal probabilities across units we use models such as the binary logistic model in this heterogeneous setting.

$\endgroup$
  • 1
    $\begingroup$ I was unclear with the fact that the probabilites of events are not identically distributed $\endgroup$ – Danny Jul 11 at 11:06
  • $\begingroup$ If the forecasts differ, the predicted probabilities need to be estimated somehow, and maximum likelihood using e.g. a binary logistic regression model is a popular way to get those estimates. $\endgroup$ – Frank Harrell Jul 12 at 17:20
3
$\begingroup$

What is wrong with your reasoning? In 1. you minimize the expectation of each term before taking the sum. You say

I Want to emphasize that each event has a different probability of occurring

and in that case, well, it is correct to minimize each term individually, but you have one parameter for each observation. Logistic regression means to model those $p_i$'s by letting them depend on some covariates, you haven't told us about covariates. Lacking them, you can do no better than the rather unhelpful estimates of 0 or 1.

Question 2. is potentially more interesting. You ask

should I interpret the Brier score such that if I minimized the expected Brier score then I am making more accurate predictions?

Well, the Brier score is just one of many other proper scoring rules and then the question becomes which proper scoring rule gives "best" predictions? Then of course you would need to define what you mean by "more accurate predictions". One paper asking this question is Choosing a Strictly Proper Scoring Rule. One could also ask if there is a theory of using proper scoring rules as inference functions, one paper is Minimum Scoring Rule Inference.

$\endgroup$
  • $\begingroup$ If I change so that $f_i = \hat{p_i}$ which was estimated by a logistic regression model then $\hat{p_i}= p_i$ still minimizes expected BS but not the sample mean which still needs $\hat{p_i}$ to be either 1 or 0 to minimize. What is the difference? @kjetil b halvorsen $\endgroup$ – Danny Jul 11 at 11:50
  • $\begingroup$ Can you please add to the question a specification of your logistic regression model? Assuming it has less parameters than observations, it will impose some structure on the $p_i$'s, and that must be takes into account. Lets say you have two groups, so a LR model with two parameres, saying that $p_i$'s are constant in each group. Then you must do the mean separately be each group, not on the individuals. $\endgroup$ – kjetil b halvorsen Jul 11 at 12:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.