11
$\begingroup$

Both log-likelihood and Brier score are proper scoring rules. As such, they reach the optimum when the predicted probabilities match the true ones. Since there is only one true probability for each predictor vector ($\textbf{x}$), minimising either of the two (or, for that matter, any other proper scoring rule) should lead to the true model - assuming the model is correctly specified, right? And if the model is not correctly specified, no scoring rule can lead to the true model, anyway.

So, if my reasoning above is correct, the optima for log-likelihood and Brier score should coincide, despite them having completely different forms. For example, log-likelihood can go into infinity, while Brier score plateaus. If they don't coincide, it indicates that the model was not correct. Is there anything to learn about the mismatch between the model and the data from analysing the differences between the models obtained by optimising for different scores?

But, isn't optimising Brier score simply least squares on a function bound to $[0, 1]$? Shouldn't we avoid least squares for probabilities, due to heteroscedasticity (and non-normality of the errors)? What am I missing?

$\endgroup$
3
  • 3
    $\begingroup$ Here is a comparison between the log and the Brier scores that may be helpful: Why is LogLoss preferred over other proper scoring rules? $\endgroup$ Mar 1 at 13:01
  • 2
    $\begingroup$ Your reasoning is correct -- in the limit, optimizing any proper scoring rule leads to the correct model, assuming it's in your class. $\endgroup$
    – usul
    Mar 2 at 2:13
  • 2
    $\begingroup$ "If they [the minima of the scores] don't coincide, it indicates that the model was not correct." Remember that proper scoring rules are minimized in expectation by the true probabilities, and that empirically calculated scores are just estimates of the expectations. So different minima may just be due to sampling variability. Finite-sample theory for proper scoring rules is not well developed. (Plus the question of course remains whether your instances all do have the same class membership probability in the first place.) $\endgroup$ Mar 2 at 17:37

1 Answer 1

16
$\begingroup$

Although both log-loss and Brier scores provide proper scoring rules, they put emphasis on different regions of the probability distributions. Quoting from Wikipedia:

the choice of a scoring rule corresponds to an assumption about the probability distribution of decision problems for which the predicted probabilities will ultimately be employed, with for example the quadratic loss (or Brier) scoring rule corresponding to a uniform probability of the decision threshold being anywhere between zero and one.

In contrast, log-loss puts a lot of emphasis on probability extremes. As you recognize, a finite data set will not give you a completely "true" model, so the choice among scoring rules is best determined by the probability cutoff region that you will use in practice, which in turn is determined by the relative costs of different types of misclassification (if you ultimately need to make classification decisions). See this page and its links for more detail.

$\endgroup$
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.