I am a non-mathematical R programmer who is completely new to the idea of scoring rules. I would like to start using them instead of classification evaluation measures like accuracy and recall, which I have recently learned are improper in terms of scoring rules. (See comments on my question Appropriate naive benchmark for class recall in binary classification for unbalanced dataset)
On one hand, it is easy to use my existing knowledge to compare models: the model with the better score (which could be high or low, depending on the scoring rule) is preferred to the other. On the other hand, a key aspect that I am missing is the notion of a benchmark of what is a good predictive model on its own without reference to other models. By comparison, when I use accuracy as a measure for classification, for a model to be considered good or useful, it must have an accuracy higher than the prevalence of the modal (most frequent) class. For instance, if there are classes A (25%), B (40%) and C (35%), then a good model must have an accuracy superior to 40%. However, I have not found any explanation of any strictly proper scoring rule that provides either such a comparable benchmark for evaluating if a score, on its own without reference to the scores of other models, is "good" or "useful".
Since the most popular scoring rules seem to be Brier (quadratic), logarithmic and spherical, could someone please give me the baseline naive benchmarks for evaluating models scored by each of these rules? (Benchmarks for other good rules would also be welcome.) And very importantly, could you please give a non-mathematical, intuitive explanation for each of these benchmarks?
Examples of the kinds of explanations I am looking for:
- For classification accuracy, the benchmark is the prevalence of the modal class because a naive classifier could attain that accuracy by simply classifying all observations to the modal class (like 40%) in the example above.
- For numeric predictions in regression, the benchmark for root square mean error (RMSE) as an error measure is the standard deviation (SD) because RMSE is the standardized variation around the prediction whereas SD is the natural standardized variation of the target variable around its mean, with an analogous mathematical formula.
Equations are fine in your explanation, but please also give the explanation in intuitive words because I do not understand complex mathematical equations.