Non-mathematical explanation of how to interpret and evaluate scoring rules in R

Question

I am a non-mathematical R programmer who is completely new to the idea of scoring rules. I would like to start using them instead of classification evaluation measures like accuracy and recall, which I have recently learned are improper in terms of scoring rules.

As a programmer, I work with statistics entirely from an input-output perspective. On the input side, I study to understand the necessary input parameters and options for a statistical procedure and, crucially, the assumptions under which the procedure is appropriate, including the necessary paramter adjustments under various circumstances. On the output side, I study to understand the meaning of the results provided, how to interpret them, and crucially, how to translate these results into real-world decision making. I do not have the mathematical foundations to understand the internal workings of the procedures, so I mostly gloss over such explanations. This approach serves me quite well.

I think I do understand the general notion: in my lay terms, I understand a strictly proper scoring rules to be a mathematical function that receives two key inputs: a set of probabilities of event outcomes; and a set of actual occurrences of the predicted events. A strictly proper scoring rule returns as output a score. When comparing the scores of two different forecasters, a strictly proper scoring rule always gives the better score (which could be high or low, depending on the scoring rule) to the forecaster that is more correct than the other.

Although there are existing R packages that implement scoring rules (https://cran.r-project.org/web/packages/scoringRules/index.html and https://cran.r-project.org/web/packages/scoring/index.html), I am unable to find any non-mathematical tutorial on how to use them meaningfully. For an example of the kind of thing I am looking for, here is a resource from which I learnt ordinal regression in R: Ordinal Logistic Regression | R Data Analysis Examples.

My question is: is there any such resource that I could use to learn how to implement and interpret scoring rules in R?

Answer 1

Scoring rules are cost functions that evaluate how far probabilistic predictions deviate from observations. Proper scoring rules have minimum cost when true probabilities are reported; strictly proper scoring rules have minimum cost only when true probabilities are reported.

The choice of scoring rule is closely connected to "how to translate these results into real-world decision making." If misclassification costs are scaled so that the cost of a false positive is $c$ and that of a false negative is $(1-c)$, then the minimum-cost cutoff for classification is at a probability of $c$.*

The Wikipedia section on Interpretation of proper scoring rules is worth quoting in detail, as it shows how this idea extends over the entire probability range to construct proper scoring rules:

All proper scoring rules are equal to weighted sums ... of the losses in a set of simple two-alternative decision problems that use the probabilistic prediction, each such decision problem having a particular combination of associated cost parameters for false positive and false negative decisions. A strictly proper scoring rule corresponds to having a nonzero weighting for all possible decision thresholds. Any given proper scoring rule is equal to the expected losses with respect to a particular probability distribution over the decision thresholds; thus the choice of a scoring rule corresponds to an assumption about the probability distribution of decision problems for which the predicted probabilities will ultimately be employed...

So the choice of scoring rule comes down to how much weight you want to place on different portions of the probability scale or, equivalently, at different relative false-positive and false-negative costs. For example, the log-loss used in logistic regression puts a lot of weight on correct probabilities at the extremes. That might not always be the best choice.

I've found "Loss Functions for Binary Class Probability Estimation and Classification: Structure and Applications," by Andreas Buja, Werner Stuetzle, and Yi Shen (2005) to be very helpful in thinking through these issues. Some of it is perhaps a bit more heavily mathematical than this question asks for, but there is enough non-mathematical content to make the fundamental issues pretty clear.

Those authors show how to fit linear models with proper scoring rules via iteratively reweighted least squares. They also describe a family of scoring rules related to beta functions that can differentially weight regions of the probability scale, and how several scoring rules--boosting loss, log loss, squared error (Brier score)--and even misclassification error (as a limiting case) can be expressed within that family.

That family is implemented in the scoring package to which you link, although perhaps not very clearly explained there (it's a fam option in the calcscore() function). There is a less mathematical, more application-oriented presentation by E. C. Merkle (package author) and M. Steyvers, "Choosing a Strictly Proper Scoring Rule," Decision Analysis Vol. 10, No. 4, December 2013, pp. 292–304. That paper explains other families of scoring rules implemented in the package, and should be a good starting point for study.

---

*That answer also shows how to incorporate gains from correct classifications into this schema, following O'Brien et al, "Cost-sensitive Multi-class Classification from Probability Estimates," Proceedings of the 25th International Conference on Machine Learning, Helsinki, Finland, 2008.