Naive benchmarks for scoring rules

Question

I am a non-mathematical R programmer who is completely new to the idea of scoring rules. I would like to start using them instead of classification evaluation measures like accuracy and recall, which I have recently learned are improper in terms of scoring rules. (See comments on my question Appropriate naive benchmark for class recall in binary classification for unbalanced dataset)

On one hand, it is easy to use my existing knowledge to compare models: the model with the better score (which could be high or low, depending on the scoring rule) is preferred to the other. On the other hand, a key aspect that I am missing is the notion of a benchmark of what is a good predictive model on its own without reference to other models. By comparison, when I use accuracy as a measure for classification, for a model to be considered good or useful, it must have an accuracy higher than the prevalence of the modal (most frequent) class. For instance, if there are classes A (25%), B (40%) and C (35%), then a good model must have an accuracy superior to 40%. However, I have not found any explanation of any strictly proper scoring rule that provides either such a comparable benchmark for evaluating if a score, on its own without reference to the scores of other models, is "good" or "useful".

Since the most popular scoring rules seem to be Brier (quadratic), logarithmic and spherical, could someone please give me the baseline naive benchmarks for evaluating models scored by each of these rules? (Benchmarks for other good rules would also be welcome.) And very importantly, could you please give a non-mathematical, intuitive explanation for each of these benchmarks?

Examples of the kinds of explanations I am looking for:

• For classification accuracy, the benchmark is the prevalence of the modal class because a naive classifier could attain that accuracy by simply classifying all observations to the modal class (like 40%) in the example above.
• For numeric predictions in regression, the benchmark for root square mean error (RMSE) as an error measure is the standard deviation (SD) because RMSE is the standardized variation around the prediction whereas SD is the natural standardized variation of the target variable around its mean, with an analogous mathematical formula.

Equations are fine in your explanation, but please also give the explanation in intuitive words because I do not understand complex mathematical equations.

Answer 1

(Proper) scoring rules assess probabilistic predictions, i.e., full continuous or discrete predictive distributions in the numerical case, and predictive class membership probabilities in the (possibly multiclass) classification case.

Specifically in the numerical case, you might be predicting tomorrow's temperatures, or sales. Your predictive distribution will be a probability density. For instance, your probabilistic temperature forecast might be "a normal distribution with mean 20°C and standard deviation 10°C", and your probabilistic sales forecast might be "a Poisson distribution with mean 3.7 units". You can then assess the actually observed temperature or sales against these probabilistic predictions using proper scoring rules, like the log score.

As a benchmark, we use the simplest reasonable model. If our complicated model cannot even beat this simple model, we have nothing to show. In your two examples, you evaluate point predictions, and the benchmarks, i.e., the simplest models, used are:

• For classification accuracy, the simplest reasonable model is to assign every new instance to the majority class of the training data.

• For numerical predictions assessed through the RMSE, the simplest model is to predict the mean of the training data. (If you wanted to minimize the Mean Absolute Error, you would instead use the median of the training data.

(Note in both cases how the error measure influences what the "best" point prediction is.)

So, what is the simplest reasonable model for probabilistic predictions?

It's what is called the climatological model: we issue a probabilistic prediction that is simply the distribution observed in the training data.

• For a classification task, the predicted probabilities would be the incidence of the classes in the training sample.

• For a numerical prediction, this would be the simple historical histogram or a density estimate (possibly smoothed).

Of course, this nomenclature comes from meteorology: your weather forecast should be at least as good as the climatological one, i.e., the multi-year average (e.g., Mason, 2004).

For references, I often recommend Tilmann Gneiting's papers. Gneiting & Katzfuss (2014) is a good overview of probabilistic predictions and proper scoring rules. Gneiting, Balabdaoui & Raftery (2007) have a nice little example comparing the climatological forecaster to more skillful colleagues. Gneiting also has a number of papers in journals like JASA and JRSS, but these are naturally more mathematical. For the discrete case (predicting counts), you may want to look at Czado, Gneiting & Held (2009), and I have published an application to (count) sales forecasting in Kolassa (2016).