2
$\begingroup$

I am a non-mathematical R programmer who is completely new to the idea of scoring rules. I would like to start using them instead of classification evaluation measures like accuracy and recall, which I have recently learned are improper in terms of scoring rules. (See comments on my question Appropriate naive benchmark for class recall in binary classification for unbalanced dataset)

On one hand, it is easy to use my existing knowledge to compare models: the model with the better score (which could be high or low, depending on the scoring rule) is preferred to the other. On the other hand, a key aspect that I am missing is the notion of a benchmark of what is a good predictive model on its own without reference to other models. By comparison, when I use accuracy as a measure for classification, for a model to be considered good or useful, it must have an accuracy higher than the prevalence of the modal (most frequent) class. For instance, if there are classes A (25%), B (40%) and C (35%), then a good model must have an accuracy superior to 40%. However, I have not found any explanation of any strictly proper scoring rule that provides either such a comparable benchmark for evaluating if a score, on its own without reference to the scores of other models, is "good" or "useful".

Since the most popular scoring rules seem to be Brier (quadratic), logarithmic and spherical, could someone please give me the baseline naive benchmarks for evaluating models scored by each of these rules? (Benchmarks for other good rules would also be welcome.) And very importantly, could you please give a non-mathematical, intuitive explanation for each of these benchmarks?

Examples of the kinds of explanations I am looking for:

  • For classification accuracy, the benchmark is the prevalence of the modal class because a naive classifier could attain that accuracy by simply classifying all observations to the modal class (like 40%) in the example above.
  • For numeric predictions in regression, the benchmark for root square mean error (RMSE) as an error measure is the standard deviation (SD) because RMSE is the standardized variation around the prediction whereas SD is the natural standardized variation of the target variable around its mean, with an analogous mathematical formula.

Equations are fine in your explanation, but please also give the explanation in intuitive words because I do not understand complex mathematical equations.

$\endgroup$

1 Answer 1

2
$\begingroup$

(Proper) scoring rules assess probabilistic predictions, i.e., full continuous or discrete predictive distributions in the numerical case, and predictive class membership probabilities in the (possibly multiclass) classification case.

Specifically in the numerical case, you might be predicting tomorrow's temperatures, or sales. Your predictive distribution will be a probability density. For instance, your probabilistic temperature forecast might be "a normal distribution with mean 20°C and standard deviation 10°C", and your probabilistic sales forecast might be "a Poisson distribution with mean 3.7 units". You can then assess the actually observed temperature or sales against these probabilistic predictions using proper scoring rules, like the log score.

As a benchmark, we use the simplest reasonable model. If our complicated model cannot even beat this simple model, we have nothing to show. In your two examples, you evaluate point predictions, and the benchmarks, i.e., the simplest models, used are:

(Note in both cases how the error measure influences what the "best" point prediction is.)

So, what is the simplest reasonable model for probabilistic predictions?

It's what is called the climatological model: we issue a probabilistic prediction that is simply the distribution observed in the training data.

  • For a classification task, the predicted probabilities would be the incidence of the classes in the training sample.

  • For a numerical prediction, this would be the simple historical histogram or a density estimate (possibly smoothed).

Of course, this nomenclature comes from meteorology: your weather forecast should be at least as good as the climatological one, i.e., the multi-year average (e.g., Mason, 2004).

For references, I often recommend Tilmann Gneiting's papers. Gneiting & Katzfuss (2014) is a good overview of probabilistic predictions and proper scoring rules. Gneiting, Balabdaoui & Raftery (2007) have a nice little example comparing the climatological forecaster to more skillful colleagues. Gneiting also has a number of papers in journals like JASA and JRSS, but these are naturally more mathematical. For the discrete case (predicting counts), you may want to look at Czado, Gneiting & Held (2009), and I have published an application to (count) sales forecasting in Kolassa (2016).

$\endgroup$
9
  • $\begingroup$ Thanks, but I don'tt understand scoring rules for numeric cases. I am very new to scoring rules (I had never heard of them until you commented on another of my questions a few days ago) and my limited reading so far only discusses probabilities for predicting discrete events with usually just two or a few possible categories, not predicting specific numbers with continuous ranges. Could you please clarify me on that point? With that misunderstanding, I cannot make sense of "For a numerical prediction, this would be the simple historical histogram or a density estimate (possibly smoothed)." $\endgroup$
    – Tripartio
    Nov 11, 2020 at 13:17
  • $\begingroup$ If you are predicting a numerical value (e.g., tomorrow's temperature, or supermarket sales), your predictive distribution will be a probability density. For instance, your probabilistic temperature forecast might be "a normal distribution with mean 20°C and standard deviation 10°C", and your probabilistic sales forecast might be "a Poisson distribution with mean 3.7 units". You can then assess the actually observed temperature or sales against these probabilistic predictions using proper scoring rules, like the log score. $\endgroup$ Nov 11, 2020 at 13:21
  • $\begingroup$ Thanks again. That makes sense. Could you please link me to a tutorial or reference that discusses scoring rules for predicting numbers? Everything I've seen so far only seems to talk about predicting discrete events (or maybe because I assumed that was the case, I failed to understand the nuances of predicting numbers). $\endgroup$
    – Tripartio
    Nov 11, 2020 at 14:14
  • $\begingroup$ Sure. Are you more interested in the continuous or in the discrete case? $\endgroup$ Nov 11, 2020 at 14:17
  • $\begingroup$ I usually recommend papers by Tilmann Gneiting, which I find very well written, like Gneiting & Katzfuss (2014), or Gneiting, Balabdaoui & Raftery (2007, linked above). He also has a number of papers in JASA and the JRSS, but those are more mathematical. For the discrete (count) case Czado, Gneiting & Held (2009). $\endgroup$ Nov 11, 2020 at 14:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.