Choosing among proper scoring rules

Question

Most resources on proper scoring rules mention a number of different scoring rules like log-loss, Brier score or spherical scoring. However, they often don't give much guidance on the differences between them. (Exhibit A: Wikipedia.)

Picking the model that maximizes the logarithmic score corresponds to picking the maximum-likelihood model, which seems like a good argument for using logarithmic scoring. Are there similar justifications for Brier or spherical scoring, or other scoring rules? Why would someone use one of these rather than logarithmic scoring?

Answer 1

Why would someone use one of these rather than logarithmic scoring?

So ideally, we always distinguish fitting a model from making a decision. In Bayesian methodology, model scoring & selection should always be done using the marginal likelihood. You then use the model to make probabilistic predictions, and your loss function tells you how to act on those predictions.

Unfortunately in the real world, computational performance often dictates that we conflate the model-selection and the decision-making and so use a loss function to fit our models. This is where subjectivity in model selection creeps in, because you've got to guess just how much different kinds of mistake will cost you. The classic example is a diagnostic for cancer: overestimating someone's probability of cancer is not good, but underestimating it is much worse.

As an aside, if you're looking for guidance on how to pick a scoring rule, you might also want to look for guidance on picking a loss function or designing a utility function, as I think the literature on those two topics is a lot more voluminous.

Answer 2

(as to my intuition)

the less the difference between predicted & actual - the more precise the forecast is. Just for the purpose of maximization of differencies - different scores are used - either log (log-loss) or difference between squares (Brier score). Everything is obvious in the mathematical formulation of the score & your needs: e.g. if you consider the bigger difference between forcasted & actual value to be the bigger errorness you'd like to take into account - of course, you'll choose log-loss, if any error (independently of size) is error for your purposes - of course, you can see Brier score (in any case, Brier score is for binary categorical outcomes -- and even in multiclass classification you will use one-vs-rest binary comparison logics). Here can see figures for log-loss & Brier. And here three properties a scoring rule should have.

Sometimes for your purposes you can even choose improper scoring rules - it's up to the purpose of your classification

You can always pick any loss_function (or cost function) or even design your own appropriate for the purposes of your estimator in training step && evaluating scores in testing step is just a mathematical formulation of how good the points are forecasted. This goodness you evaluate according this mathematical formulation - chosen according the nature of outcomes (numerical or categorical, but discrete anyway for classification)

P.S. In statistics: statistical estimation should be unbiased, effective, careful. And Variance Components used in estimation include: Continuous dependent - Dependent, Categorical dependent - Random effects, Categorical predictors - Fixed effects, Continuous predictors - Covariates, .. and their interactions. So, In probability theory applied to statistically meaningful data I prefer to follow the same rules for scoring: unbiased, effective, careful -- though it is really not so easy to develop such model [for any distribution of features given, much depends on the size of sample and its inner variation for comprehensive modelling] -- applied mathematics, I think, should help in designing appropriate loss-functions & scores based on them (just for your task's objectivity & convenience).

ALSO the choice of score can depend on the nature of process you explore: either stochastic or deterministic or dynamic, I think. You should be aware of either you need probabilistic or deterministic model based on your task. With first you can input in loss (or/and score) the size of errorness, with second you need just any score for binary outcomes, with third even improper naive linear score would be enough for me sometimes

Anyway, with statistically meaningful data (feature engineered & selected correctly) modelling becomes easier & scores simplier are affordable