30
$\begingroup$

Most resources on proper scoring rules mention a number of different scoring rules like log-loss, Brier score or spherical scoring. However, they often don't give much guidance on the differences between them. (Exhibit A: Wikipedia.)

Picking the model that maximizes the logarithmic score corresponds to picking the maximum-likelihood model, which seems like a good argument for using logarithmic scoring. Are there similar justifications for Brier or spherical scoring, or other scoring rules? Why would someone use one of these rather than logarithmic scoring?

$\endgroup$
3
  • 2
    $\begingroup$ Some hints are in the nomenclature. "Cost functional" is from optimization or optimal control system engineering. There is no "best". To have a "good" means you must have a measure of goodness. There are infinite numbers of families of measures of goodness. A trivial example is: what is the best path? If you are marching to your execution - make it a long pleasant one. If you are going to your Fields metal, make it shortest. System expertise helps you select the measure of goodness. When you have the measure of goodness, then you can find "best". $\endgroup$ Dec 12, 2014 at 19:05
  • 1
    $\begingroup$ You may be interested in Merkle & Steyvers, "Choosing a strictly proper scoring rule" (2013, Decision Analysis). $\endgroup$ Mar 22, 2018 at 16:22
  • 2
    $\begingroup$ I took the liberty of editing the title to make it more precise/informative. If I misinterpreted it, sorry and feel free to revert back the change. $\endgroup$ Sep 19, 2019 at 11:30

2 Answers 2

20
+50
$\begingroup$

Why would someone use one of these rather than logarithmic scoring?

So ideally, we always distinguish fitting a model from making a decision. In Bayesian methodology, model scoring & selection should always be done using the marginal likelihood. You then use the model to make probabilistic predictions, and your loss function tells you how to act on those predictions.

Unfortunately in the real world, computational performance often dictates that we conflate the model-selection and the decision-making and so use a loss function to fit our models. This is where subjectivity in model selection creeps in, because you've got to guess just how much different kinds of mistake will cost you. The classic example is a diagnostic for cancer: overestimating someone's probability of cancer is not good, but underestimating it is much worse.

As an aside, if you're looking for guidance on how to pick a scoring rule, you might also want to look for guidance on picking a loss function or designing a utility function, as I think the literature on those two topics is a lot more voluminous.

$\endgroup$
4
  • 5
    $\begingroup$ 1) Are you saying that Brier scoring is essentially a "loss function in disguise"--that is, even though it masquerades as a utility-function-agnostic scoring/comparison rule, it's actually used because people have specific preferences over the types of errors the model makes? $\endgroup$
    – Ben Kuhn
    Dec 15, 2014 at 18:54
  • 1
    $\begingroup$ 2) Do you have any specific examples of settings in which someone might choose Brier or spherical scoring over log scoring (= marginal likelihood, as I understand it) for those reasons? $\endgroup$
    – Ben Kuhn
    Dec 15, 2014 at 18:55
  • 1
    $\begingroup$ 3) Why would it perform better to bake your loss/utility function assumptions into the model than to fit to marginal likelihood and use your loss/utility function when actually making the decision? It seems like for ideal learning algorithms there should be no gap between these. $\endgroup$
    – Ben Kuhn
    Dec 15, 2014 at 19:01
  • 6
    $\begingroup$ 1) Yep. 2) Not personally, no. Scoring rules aren't "fashionable" in the bit of ML I work in. Having a quick poke around on Scholar, it seems they're a bit dated in general. This paper looks like it'd be interesting to you though. 3) By performance I meant "computational performance", not "predictive performance". $\endgroup$
    – Andy Jones
    Dec 15, 2014 at 19:45
0
$\begingroup$

(as to my intuition)

the less the difference between predicted & actual - the more precise the forecast is. Just for the purpose of maximization of differencies - different scores are used - either log (log-loss) or difference between squares (Brier score). Everything is obvious in the mathematical formulation of the score & your needs: e.g. if you consider the bigger difference between forcasted & actual value to be the bigger errorness you'd like to take into account - of course, you'll choose log-loss, if any error (independently of size) is error for your purposes - of course, you can see Brier score (in any case, Brier score is for binary categorical outcomes -- and even in multiclass classification you will use one-vs-rest binary comparison logics). Here can see figures for log-loss & Brier. And here three properties a scoring rule should have.

Sometimes for your purposes you can even choose improper scoring rules - it's up to the purpose of your classification

You can always pick any loss_function (or cost function) or even design your own appropriate for the purposes of your estimator in training step && evaluating scores in testing step is just a mathematical formulation of how good the points are forecasted. This goodness you evaluate according this mathematical formulation - chosen according the nature of outcomes (numerical or categorical, but discrete anyway for classification)

P.S. In statistics: statistical estimation should be unbiased, effective, careful. And Variance Components used in estimation include: Continuous dependent - Dependent, Categorical dependent - Random effects, Categorical predictors - Fixed effects, Continuous predictors - Covariates, .. and their interactions. So, In probability theory applied to statistically meaningful data I prefer to follow the same rules for scoring: unbiased, effective, careful -- though it is really not so easy to develop such model [for any distribution of features given, much depends on the size of sample and its inner variation for comprehensive modelling] -- applied mathematics, I think, should help in designing appropriate loss-functions & scores based on them (just for your task's objectivity & convenience).

ALSO the choice of score can depend on the nature of process you explore: either stochastic or deterministic or dynamic, I think. You should be aware of either you need probabilistic or deterministic model based on your task. With first you can input in loss (or/and score) the size of errorness, with second you need just any score for binary outcomes, with third even improper naive linear score would be enough for me sometimes

Anyway, with statistically meaningful data (feature engineered & selected correctly) modelling becomes easier & scores simplier are affordable

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.